1Appendix for Latent Dirichlet Allocation Models for Image Classification APPENDIX I VARIATIONAL INFERENCE IN LDA MODELS

consists of computing the posterior distribution of the unob-served variables, P (pi, z1:N |I) 1. Learning involves estimating the parameters (α,Λ1:K), by maximizing the log likelihood, l = logP (D) of a training image dataset D. Inference and learning are not tractable under LDA. A wide range of approx-imate inference methods have been proposed, such as Laplace or variational approximations, sampling methods, etc. We adopt variational inference. Variational methods approximate the posterior P (pi, z1:N |w1:N) by a mean-field variational distribution q(pi, z1:N), indexed by free variational parameters, within some class of tractable probability distributions F. These distributions usually assume independent factors, q(pi, z1:N) = q(pi;γ) n q(zn;φn) (1) where q(y) and q(zn) are categorical models, and q(pi) a Dirichlet distribution. Given an observation w1:N, the opti-mal variational approximation minimizes the Kullback-Leibler (KL) divergence between the two posteriors q ∗ = argmin q∈F KL(q(pi, z1:N)||P (pi, z1:N |w1:N)) (2) = argmin q∈F logP (w1:N) − L(γ,φ;α,Λ) (3) where L(γ,φ;α,Λ) = Eq[logP (pi, z1:N, w1:N;α,Λ1:K)] − Eq[log q(pi, z1:N;γ,φ1:N)] (4) is commonly known as the evidence lower bound (ELBO) [1]. This also lower bounds the true log likelihood of an image, for an arbitrary variational distribution q(pi, z1:N) (see [2], Appendix A.3). Since the data likelihood P (w1:N) is constant for an ob-served image, the optimization problem of (3) is identical to the maximization of the ELBO, q∗(pi, z1:N) = argmax q∈F L(γ,φ;α,Λ) (5) From Appendix A.3 of [2], the update rule for coordinate descent of the variational parameters is γ∗k = n φnk + αk (6