Decentralized Variational Bayesian Inference

This work presents a decentralized, approx-imate method for performing variational in-ference on a network of learning agents. The key difficulty with performing decentralized inference is that for most Bayesian mod-els, the use of approximate inference al-gorithms is required, but such algorithms destroy symmetry and dependencies in the model that are crucial to properly combining the local models from each individual learn-ing agent. This paper first investigates how approximate inference schemes break depen-dencies in Bayesian models. Using insights gained from that investigation, an optimiza-tion problem is proposed whose solution ac-counts for those broken dependencies when combining local posteriors. Experiments on synthetic and real data demonstrate that the decentralized method provides advantages in computational performance and predictive test likelihood over previous centralized and distributed methods.