Global Optimality in Representation Learning

A majority of data processing techniques across a wide range of technical disciplines require a representation of the data that is meaningful for the task at hand in order to succeed. In some cases one has enough prior knowledge about the problem that a fixed transformation of the data or set of features can be pre-calculated, but for most challenging problems with high dimensional data, it is often not known what representation of the data would give the best performance. To address this issue, the field of representation learning seeks to learn meaningful representations directly from data and includes methods such as matrix factorization, tensor factorization, and neural networks. Such techniques have achieved considerable empirical success in many fields, but common to a vast majority of these approaches are the significant disadvantages that 1) the associated optimization problems are typically non-convex due to a multilinear form or other convexity destroying transformation and 2) one is forced to specify the size of the learned representation a priori. This thesis presents a very general framework which allows for the mathematical analysis of a wide range of non-convex representation learning problems. The framework allows the derivation of sufficient conditions to guarantee that a local minimizer of the non-convex optimization problem is a global minimizer and that from any initialization it is possible to find a global minimizer using a purely local descent algorithm. Further, the framework also allows for a wide range of regularization to be incorporated into the model to capture known features of data and to adaptively fit the size of the learned representation to the data instead of defining it a priori. Multiple implications of this work are discussed as they relate to modern practices in deep learning, and the advantages of the approach are demonstrated in applications of automated spatio-temporal segmentation of neural calcium imaging data and reconstructing hyperspectral image volumes from compressed measurements