Bayesian Manifold Learning: Locally Linear Latent Variable Model (LL-LVM)

Problems with high-dimensional data optimisation in high-d parameter space is computationally expensive and hard to find a global optimum Good news: in many cases, the intrinsic dimensionality is actually low datapoints are sampled from a low-dimensional manifold embedded in a high-dimensional space example: swiss roll Adapted from Roweis & Saul, Science, 2000 Manifold learning: attempts to uncover the manifold structure Non-probabilistic prior work idea: preserve geometric properties of local neighbourhoods limits: sensitive to noise due to lack of explicit model heuristic methods to evaluate manifold dimensionality no measure of uncertainties in the estimates out-of-sample extension requires extra approximations Gaussian process latent variable model (GP-LVM) idea: define a functional mapping from latent space to data space using GP [1, 2] for data Y = [y1,...,ydy] ∈ Rn×dy and latents X = [x1,...,xdx] ∈ Rn×dx, p(Y|X) = dy∏ k=1 N (yk|0,Knn + β−1In), where the i, jth element of the covariance matrix is k(xi,xj) = σ 2 f exp −12 dx∑ q=1 αq(xi,q − xj,q)2 where αq’s determine dimensionality of latent space. limits: no intuitive preservation of local neighbourhood properties smoothness of manifold constrained by pre-chosen covariance function auxiliary variable for variational inference (also restricts choice of cov func) Question Can we learn a manifold in a probabilistic and possibly Bayesian way, while preserving geometric properties of local neighbourhoods? Our approach: LL-LVM Key idea: there is a locally linear mapping between tangent spaces in low and high dimensional spaces high-dimensional space low-dimension space T yi yi y