Latent variable bayesian models for promoting sparsity

Abstract—Many practical methods for finding maximally sparse coefficient expansions involve solving a regression problem using a particular class of concave penalty functions. From a Bayesian per-spective, this process is equivalent to maximum a posteriori (MAP) estimation using a sparsity-inducing prior distribution (Type I esti-mation). Using variational techniques, this distribution can always be conveniently expressed as a maximization over scaled Gaussian distributions modulated by a set of latent variables. Alternative Bayesian algorithms, which operate in latent variable space lever-aging this variational representation, lead to sparse estimators re-flecting posterior information beyond the mode (Type II estima-tion). Currently, it is unclear how the underlying cost functions of Type I and Type II relate, nor what relevant theoretical proper-ties exist, especially with regard to Type II. Herein a common set of auxiliary functions is used to conveniently express both Type I and Type II cost functions in either coefficient or latent variable space facilitating direct comparisons. In coefficient space, the anal-ysis reveals that Type II is exactly equivalent to performing stan-dard MAP estimation using a particular class of dictionary- and noise-dependent, nonfactorial coefficient priors. One prior (at least) from this class maintains several desirable advantages over all pos-sible Type I methods and utilizes a novel, nonconvex approximation to the norm with most, and in certain quantifiable conditions all, local minima smoothed away. Importantly, the global minimum is always left unaltered unlike standard -norm relaxations. This en-sures that any appropriate descent method is guaranteed to locate the maximally sparse solution. Index Terms—Bayesian learning, compressive sensing, latent variable models, source localization, sparse priors, sparse repre-sentations, underdetermined inverse problems. I