Geometry of the restricted Boltzmann machine

We describe discrete restricted Boltzmann machines: probabilistic graphical mod-els with bipartite interactions between discrete visible and hidden variables. These models generalize standard binary restricted Boltzmann machines and discrete naı̈ve Bayes models. For a given number of visible variables and cardinalities of their state spaces, we bound the number of hidden variables, depending on the cardinalities of their state spaces, for which the model is a universal approximator of probability distributions. More generally, we describe exponential subfamilies and use them to bound the Kullback-Leibler approximation errors of these models from above. We use coding theory and algebraic methods to study the geometry of these models, and show that in many cases they have the dimension expected from counting parameters, but in some cases they do not. We discuss inference func-tions, mixtures of product distributions with shared parameters, and patterns of strong modes of probability distributions represented by discrete restricted Boltz-mann machines in terms of configurations of projected products of simplices in normal fans of products of simplices.