Objective function

Each data point is generated according to the following process: 1. Gaussian distribution k is chosen according to pk 2. data point x is generated according to Gaussian with parameters Σ k, µ kWhat is the EM algorithm? A maximum likelihood estimation technique for problems with hidden variables. ◮ Maximum likelihood estimation: where: ◮ X: observered data ◮ θ: parameters ˆθ = arg max log P(X |θ) θ ◮ hidden variable: choice in generative model that cannot be observed. Hidden Variables Generative models ◮ typical model without hidden variables: 1. datapoint x (observable) is generated according to parameters θ. ◮ typical model with hidden variables: 1. a value for hidden variable Y is chosen according to θ. 2. datapoint x (observable) is generated according to parameters θ and choice for Y.Hidden Variables What is the hidden variable in a Gaussian mixture model? Hidden Variables What is the hidden variable in a Gaussian mixture model? 1. Gaussian distribution k is chosen according to pk 2. data point x is generated according to Gaussian with parameters Σ k, µ k ⇒ Which gaussian distribution generated a data point. Parameters What are the parameters of a Gaussian mixture model? Parameters What are the parameters of a Gaussian mixture model? 1. Gaussian distribution k is chosen according to pk 2. data point x is generated according to Gaussian with parameters Σ k, µ kInference Problems for Hidden Variable Models ◮ Given a data point x, what is the probability that the choice for the hidden variable was y. ◮ Given the data, what are parameters that give maximu