Annealing between distributions by averaging moments

Many powerful Monte Carlo techniques for estimating partition functions, such as annealed importance sampling (AIS), are based on sampling from a sequence of intermediate distributions which interpolate between a tractable initial distribu-tion and the intractable target distribution. The near-universal practice is to use geometric averages of the initial and target distributions, but alternative paths can perform substantially better. We present a novel sequence of intermediate distribu-tions for exponential families defined by averaging the moments of the initial and target distributions. We analyze the asymptotic performance of both the geomet-ric and moment averages paths and derive an asymptotically optimal piecewise linear schedule. AIS with moment averaging performs well empirically at esti-mating partition functions of restricted Boltzmann machines (RBMs), which form the building blocks of many deep learning models.