---

Apprentissage de données fonctionnelles par modèles multi-tâches :
Application à la prédiction de performances sportives

---

Origins of the thesis

A problem:

• Several papers (Boccia & al - 2017, Kearney & Hayes - 2018) point out limits of focusing on best performers in young categories.
• Sport experts seek new objecive criteria for talent identification.

An opportunity:

• The French Swimming Federation (FFN) provides a massive database gathering most of the national competition's results since 2002.

Data

Performances in competition for FFN members:

Data

Performances in competition for FFN members:

• Irregular time series (in number of observations and location),
• Many different swimmers per category,

Data

Performances in competition for FFN members:

• Irregular time series (in number of observations and location),
• Many different swimmers per category,
• A few observations per swimmer.

Contributions

Are there different patterns of progression among swimmers?
Chapter 2
Leroy et al. - Functional Data Analysis in Sport Science: Example of Swimmers’ Progression Curves Clustering - Applied Sciences - 2018

Can we provide probabilistic predictions of future performances?
Chapter 3
Leroy et al. - Magma: Inference and Prediction with Multi-Task Gaussian Processes - Under submission in Statistics and Computing - https://github.com/ArthurLeroy/MAGMA

May possible group structures improve the quality of predictions?
Chapter 4
Leroy et al. - Cluster-Specific Predictions with Multi-Task Gaussian Processes - Under submission in JMLR - https://github.com/ArthurLeroy/MAGMAclust

Contributions

Are there different patterns of progression among swimmers?

Common representation and curve clustering

A common representation as functional data is proposed by using B-splines decomposition.

Clustering curves with FunHDDC algorithm (Bouveyron & Jacques - 2011, Schmutz et al. - 2020) highlights different patterns of progression. These groups, relating both on level and trend, are consistent groups with sport experts knowledge.

Limits in terms of modelling

This approach suffers from severe limitations such as:

• unsatisfying individual modellings (side effects, sensibility to sparsity, ...),
• a lack of probabilistic prediction methods,
• persisting troubles with irregular measurements.

Contributions

Can we provide probabilistic predictions of future performances?

Gaussian process regression

No restrictions on f but a prior distribution on a functional space: f∼GP(0,C(⋅,⋅))

Modelling and prediction with a unique GP

GPs provide an ideal framework for modelling although insufficient for direct predictions.

Multi-task GP with common mean (Magma)

with:

Notation and dimensionality

Each individual has its specific vector of inputs ti associated with outputs yi.
The mean process μ0 requires to define pooled vectors and additional notation follows:

EM algorithm: E step

with:

EM algorithm: M step

EM algorithm: M step

Prediction

For a new individual, we observe some data y∗(t∗). Let us recall:

Goals:

Prediction: the key idea

Defining a multi-task prior distribution by:

• conditioning on training data,
• integrating over μ0's hyper-posterior distribution.

with:

Prediction: additional steps

A picture is worth 1000 words

• Same data, same hyper-parameters
• Standard GP (left), Magma (right)

A GIF is worth 109 words

Illustration: GP regression

Contributions

May eventual group structures improve the quality of predictions?

Magma + Clustering = MagmaClust

A unique underlying mean process might be too restrictive.

→ Mixture of multi-task GPs:

with:

• Zi∼M(1,π), ⊥⊥i,

• μ0∼GP(m0,Kθ0), ⊥⊥k,
• fi∼GP(0,Σθi), ⊥⊥i,
• ϵi∼GP(0,σ2i), ⊥⊥i.

It follows that:

Magma + Clustering = MagmaClust

A unique underlying mean process might be too restrictive.

→ Mixture of multi-task GPs:

with:

• Zi∼M(1,π), ⊥⊥i,
• μk∼GP(mk,Cγk) ⊥⊥k,
• fi∼GP(0,Σθi), ⊥⊥i,
• ϵi∼GP(0,σ2i), ⊥⊥i.

It follows that:

Magma + Clustering = MagmaClust

A unique underlying mean process might be too restrictive.

→ Mixture of multi-task GPs:

with:

• Zi∼M(1,π), ⊥⊥i,
• μk∼GP(mk,Cγk) ⊥⊥k,
• fi∼GP(0,Σθi), ⊥⊥i,
• ϵi∼GP(0,σ2i), ⊥⊥i.

It follows that:

Learning

The integrated likelihood is not tractable anymore due to posterior dependencies between μ={μk}k and Z={Zi}i.

Variational inference still allows us to maintain closed-form approximations. For any distribution q: logp(y∣Θ)=L(q;Θ)+KL(q∣∣p(μ,Z∣y,Θ))

The posterior independance is forced by an approximation assumption: q(μ,Z)=qμ(μ)qZ(Z).

Maximising the lower bound L(q;Θ) induces natural factorisations over clusters and individuals for the variational distributions.

Variational EM: E step

with:

Variational EM: M step

Covariance structure assumption: 4 sub-models

Sharing the covariance structures offers a compromise between flexibility and parsimony:

• Hoo: common mean process - common individual process - 2 HPs,
• Hko: specific mean process - common individual process - K+1 HPs,
• Hoi: common mean process - specific individual process - M+1 HPs,
• Hki: specific mean process - specific individual process - M+K HP.

Prediction

• EM for estimating p(Z∗∣y,ˆΘ), ˆθ∗, and ˆσ2∗,
• Multi-task prior: p([y∗(t∗)y∗(tp)]∣Z∗k=1,y)=N([y∗(t∗)y∗(tp)];[ˆmk(t∗)ˆmk(tp)],(Γk∗∗Γk∗pΓkp∗Γkpp)),∀k,
• Multi-task posterior:

  p(y∗(tp)∣y∗(t∗),Z∗k=1,y)=N(ˆμk∗(tp),ˆΓkpp), ∀k,

• Predictive multi-task GPs mixture:

  p(y∗(tp)∣y∗(t∗),y)=K∑k=1τ∗k N(ˆμk∗(tp),ˆΓkpp(tp)).

Illustration: Magma vs MagmaClust

Clustering performances

Predictive performances on swimmers datasets

And what about the swimmers ?

And what about the swimmers ?

And what about the swimmers ?

Predictive performances on swimmers datasets

Did I mention that I like GIFs ?

Perspectives: model selection in MagmaClust

Leroy & al. - 2020 - preprint

After convergence of the VEM algorithm, a variational-BIC expression can be derived as:

Perspectives

• Enable association with sparse GP approximations,
• Extend to multivariate functional regression,
• Develop an online version,
• Integrate to the FFN app and launch real-life tests,
• Investigate GP variational encoder for functional data

References

• Neal - Priors for infinite networks - University of Toronto - 1994
• Shi and Wang - Curve prediction and clustering with mixtures of Gaussian process functional regression models - Statistics and Computing - 2008
  
• Bouveyron and Jacques - Model-based clustering of time series in group-specific functional subspaces - Advances in Data Analysis and Classification - 2011
• Boccia et al. - Career Performance Trajectories in Track and Field Jumping Events from Youth to Senior Success: The Importance of Learning and Development - PLoS ONE - 2017
• Lee et al. - Deep Neural Networks as Gaussian Processes - ICLR - 2018
  
• Kearney et Hayes - Excelling at youth level in competitive track and field athletics is not a prerequisite for later success - Journal of Sports Sciences - 2018
• Leroy et al. - Functional Data Analysis in Sport Science: Example of Swimmers’ Progression Curves Clustering - Applied Sciences - 2018
  
• Schmutz et al. - Clustering multivariate functional data in group-specific functional subspaces - Computational Statistics - 2020
  
• Leroy et al. - Magma: Inference and Prediction with Multi-Task Gaussian Processes - Under review - 2020
• Leroy et al. - Cluster-Specific Predictions with Multi-Task Gaussian Processes - Under review - 2020

Annexe

Two important prediction steps of Magma have been omitted for clarity:

• Recomputing the hyper-posterior distribution on the new grid: p(μ0(tp∗)∣y),
• Estimating the hyper-parameters of the new individual: ˆθ∗,ˆσ2∗=argmaxθ∗,σ2∗ p(y∗(t∗)∣y,θ∗,σ2∗).

The computational complexity for learning is given by:

• Magma: O(M×N3i+N3)
• MagmaClust: O(M×N3i+K×N3)

Annexe: MagmaClust, remaining clusters

Annexe: model selection performances of VBIC