Apprentissage multi-tâches de données fonctionnelles :
Applications à la détection pour le sport de haut niveau

Context

Traditional talent identification:
→ Best young athlete + coach intuition

G. Boccia et al. (2017) :

≃ 60% of 16 years old elite athletes do not maintain their level of performance

Philip E. Kearney & Philip R. Hayes (2018) :

≃ only 10% of senior top 20 were also top 20 before 13 years

Data

Performances from FF of Swimming members since 2002:

• Irregular time series
• Different number Ni of observations between individuals
• Different observational timestamps tki
• Ni ≃x×101

Data

Performances from FF of Swimming members since 2002:

• Irregular time series
• Different number Ni of observations between individuals
• Different observational timestamps tki
• Ni ≃x×101 | N =M∑i=1 Ni ≃x×105

Data

Performances from FF of Swimming members since 2002:

• Irregular time series
• Different number Ni of observations between individuals
• Different observational timestamps tki
• Ni ≃x×101 | N =M∑i=1 Ni ≃x×105

Curves clustering

Functional data ≃ coefficients αk of B-splines functions:

yi(t)=K∑k=1αkBk(t)

Clustering: Algo FunHDDC (gaussian mixture + EM)
Bouveyron & Jacques - 2011

Using the multidimensional version : curve + derivative
→ Information about performance level and trend of improvement

Curve clustering

Leroy et al. - 2018

• Different patterns of progression
• Consistent groups for sport experts

Curve clustering

Leroy et al. - 2018

• Different patterns of progression
• Consistent groups for sport experts

New objectives

• Prediction of the future values of the progression curve
  → Functional regression
• Quantification of prediction uncertainty
  → Probabilistic framework

Gaussian process regression

Bishop - 2006 | Rasmussen & Williams - 2006

GPR : a kernel method to estimate f when:

y=f(x)+ϵ

→ No restrictions on f but a prior probability:

f∼GP(0,C(⋅,⋅))

An example of exponential kernel for the covariance function: cov(f(x),f(x′))=C(x,x′)=αexp(−12θ2|x−x′|2) Kernel definition ⇒ prefered properties on f

Prediction

yN+1=(y1,...,yN+1) has the following prior density: yN+1∼N(0,CN+1), CN+1=(CNkN+1kTN+1cN+1)

When the joint density is gaussian, so does the conditionnal dentisty:

yN+1|yN,xN+1∼N(kTC−1NyN,cN+1−kTN+1C−1NkN+1)

• Prediction: ˆyN+1=E[yN+1|yN,xN+1]
• Uncertainty: CI with V[yN+1|yN,xN+1]

Visualization of GPR

Key points:

• Define a covariance function with desirable properties
• Non parametric method giving probabilistic predictions
• Complexity O(N3) (inversion of a N×N matrix)

GP estimation from data

Estimating a GP on each individuals (O(N3i)):

• Uncertainty: Ok

GP estimation from data

Estimating a GP on each individuals (O(N3i)):

• Uncertainty: Ok

GP estimation from data

Estimating a GP on each individuals (O(N3i)):

• Uncertainty: Ok

GP estimation from data

Estimating a GP on each individuals (O(N3i)):

• Uncertainty: Ok

GP estimation from data

Estimating a GP on each individuals (O(N3i)):

• Uncertainty: Ok

Reaching a coherent modeling

Estimating a GP on each individuals (O(N3i)):

• Uncertainty: Ok
• Coherence: Improvement required

→ Using the shared information between individuals (GPR-ME)

The GPFR model

Shi & Wang - 2008 | Shi & Choi - 2011

yi(t)=μ0(t)+fi(t)+ϵi with:

• μ0(t)=K∑k=1αkBk(t) with (Bk)k a spline basis
• fi(⋅)∼GP(0,Σθi(⋅,⋅)), fi⊥⊥
• ϵi∼N(0,σ2), ϵi⊥⊥

GPFDA R package

Limits:

• No uncertainty about μ0
• Does not allow irregular time series

An extension to GPFR

yi(t)=μ0(t)+fi(t)+ϵi with:

• μ0(⋅)∼GP(m0,Kθ0(⋅,⋅))
• fi(⋅)∼GP(0,Σθi(⋅,⋅)), fi⊥⊥
• ϵi∼N(0,σ2i), ϵi⊥⊥

It follows that:

yi(⋅)|μ0∼GP(μ0(⋅),Σθi(⋅,⋅)+σ2), yi|μ0⊥⊥

→ Shared information through μ0 and its uncertainty
→ Unified non parametric probabilistic framework
→ Effective even for irregular time series

Notations

y=(y11,…,yki,…,yNMM)T
t=(t11,…,tki,…,tNMM)T
Θ={θ0,(θi)i,σ2i}

Σ: covariance matrix from the process fi evaluated on t

Σ=[Σθi(tki,tlj)(i,j),(j,l)]

Ψ=Σ+σ2iIdN

Bayes’ law is the new Bible

Reminder of its simple definition:

P(T|D)=P(D|T)P(T)P(D) Powerful implication when it comes to learning from data:

• P(T), probability of your theory, what you think a priori
• P(D|T), probability of data if theory is true, likelihood
• P(D), probability that your data occur, norm. constant

Bayes’ law tells you how and what you should learn on theory T according to data D:

→P(T|D), what you should think a posteriori Computational burden, among solutions: empirical Bayes

Learning HP and μ0 : an EM algorithm

Step E: Computing the posterior (knowing Θ)

p(μ0(t)|t,y,Θ)∝p(y|t,μ0(t),Θ) p(μ0(t)|t,Θ)∝N(μ0(t),Ψ) N(m0,K)=[Insert here some PhD student ideas]=N(ˆμ0(t),ˆK)

Step M: Estimating Θ (knowing p(μ0))

ˆΘ=argmaxΘ Eμ0[log p(y,μ0(t)|t,Θ) |Θ]

    Initialize hyperparameters
    while(sufficient condition of convergence){
    Iterate alternatively steps E and M}

A picture is worth 1000 words

E[μ0(t)|Data]±CI0.95

Iteration counter : 0

A picture is worth 1000 words

E[μ0(t)|Data]±CI0.95

Iteration counter : 1

A picture is worth 1000 words

E[μ0(t)|Data]±CI0.95

Iteration counter : 2

A picture is worth 1000 words

E[μ0(t)|Data]±CI0.95

Iteration counter : 4

A picture is worth 1000 words

E[μ0(t)|Data]±CI0.95

Iteration counter : 5

A picture is worth 1000 words

E[μ0(t)|Data]±CI0.95

Iteration counter : 6 → break and return

Coupure pub

Petit jeu concours pour savoir qui sont ceux qui suivent et ceux qui scrollent sur Facebook :

L’orateur a oublié de sortir un graph et a donc sauté une itération.

Saurez vous retrouver laquelle ?

→ Une bière pour le vainqueur. Un jour. Peut être.

Making predictions

∀i,  yi(t)=μ0(t)+fi(t)+ϵi Suppose that, after the learning step, you observe data from a new individual ⋆. Multitask learning consists in improving performance by sharing information across individuals.

Also recall that if p(y∗(t),y∗(tnew))=N([mt∗mnew∗],(Ψt,t∗Ψt,new∗Ψnew,t∗Ψnew,new∗))

GP prediction’s formula gives:

p(y∗(tnew)|y∗(t))=N(mnew∗+Ψnew,t∗Ψt,t∗−1(y∗(t)−mt∗);Ψnew,new∗−Ψnew,t∗Ψt,t∗−1Ψt,new∗)

Making predictions

Multitask regression : conditioning on observations Incertitude on the mean process : integrate over μ0

p(y∗|y)=∫p(y∗,μ0|y) dμ0=⏟Bayes♡∫p(y∗|y,μ0)p(μ0|y) dμ0=⏟(yi|μ0)i⊥⊥∫p(y∗|μ0)p(μ0|y) dμ0=N(ˆμ0,ˆK+Ψ)

A gif is worth 109 words

• Same data, same hyperparameters from learning
• Standard GP (left) | Our Multitask GP (right)

A gif is worth 109 words

We talked about clustering, did we ?

Assumption : 1 underlying mean process might be strong
→ Mixture model of multitask GP:

∀i,∀k,  yi(t)|(Zik=1)=μk(t)+fi(t)+ϵi with:

• Zi∼M(1,π)
• μk(⋅)∼GP(mk,Kθ0(⋅,⋅))
• fi(⋅)∼GP(0,Σθi(⋅,⋅)), fi⊥⊥
• ϵi∼N(0,σ2i), ϵi⊥⊥

It follows that:

yi(⋅)|(μk)k,π∼K∑k=1πk GP(μk(⋅),Ψi), yi|(μk)k,π⊥⊥

Learning (μk)k,(Zi)i and HPs : VEM

Posterior dependencies (μk)k and (Zi)i → variational EM

Step E:

Approximation assumption: r(μ,Z)=r(μ)p(Z)
True likelihood becomes : L(model)=L(r(μ,Z))+KL(r(μ,Z)||p(μ,Z|y))

Step M:

L(r(μ,Z)) provides a lower bound for LL maximization
Then buisiness as usual

→ Each step is proved to increase likelihood. Repeat until convergence.

Did I mention that I like gifs ?

Why probabilistic prediction is better ?

Making a prediction is P(saying something wrong)≃1.
A probabilistic prediction tells you how much:

Perspectives

• Improve and release the package

• Enable association with sparse GP approximations

• Integrate to the app and launch tests with FFN

• Maybe multivariate functional regression

• If you have another idea, let’s work on that together

• Write down a thesis. Someday.

A cat gif is priceless