Communicated by Radford Neal Statistical Physics, Mixtures of Distributions, and the EM Algorithm

We show that there are strong relationships between approaches to optmization and learning based on statistical physics or mixtures of experts. In particular, the EM algorithm can be interpreted as converg-ing either to a local maximum of the mixtures model or to a saddle point solution to the statistical physics system. An advantage of the statistical physics approach is that it naturally gives rise to a heuristic continuation method, deterministic annealing, for finding good solu-tions. In recent years there has been considerable interest in formulating op-timization problems in terms of statistical physics. This has led to the development of powerful optimization algorithms, such as deterministic annealing. At the same time good results have been attained by formulating learning theory in terms of mixtures of distributions (Jacobs et al. 1991) and using the EM algorithm (Jordan and Jacobs 1993). The aim of this note is to show that there are close connections be-tween the mixture of distributions and the statistical physics approaches. The EM algorithm can be, and has been, used in conjunction with deter-ministic annealing. This equivalence has previously been mentioned fo