Sparse Structure Learning via Information-Theoretic Regularization and Self-Contained Probabilistic Estimation

Nowadays, there is an increasing amount of digital information constantly generated from every aspect of our life and data that we work with grow in both size and variety. Fortunately, most of the data have sparse structures. Compressive sensing offers us an efficient framework to not only collect data but also to process and analyze them in a timely fashion. Various compressive sensing tasks eventually boil down to the sparse signal recovery problem in an under-determined linear system. To better address the challenges of ``big'' data using compressive sensing, we focus on developing powerful sparse signal recovery approaches and providing theoretical analysis of their optimalities and convergences in this dissertation. Specifically, we bring together insights from information theory and probabilistic graphical models to tackle the sparse signal recovery problem from the following two perspectives: Sparsity-regularization approach: we propose the Shannon entropy function and Renyi entropy function constructed from the sparse signal, and prove that minimizing them does promote sparsity in the recovered signal. Experiments on simulated and real data show that the two proposed entropy function minimization methods outperform state-of-the-art lp-norm minimization and l1-norm minimization methods. Probabilistic approach: we propose the generalized approximate message passing with built-in parameter estimation (PE-GAMP) framework, present its empirical convergence analysis and give detailed formulations to obtain the MMSE and MAP estimations of the sparse signal. Experiments on simulated and real data show that the proposed PE-GAMP is more robust, much simpler and has a wider applicability compared to the popular Expectation Maximization based parameter estimation method