Storage Capacity of a fully-connected parity machine with continuous weights

We study a fully-connected parity machine with K hidden units for continuous weights. The geometrical structure of the weight space of this model is analyzed in terms of the volumes associated with the internal representations of training set. By examining the asymptotic behavior of order parameters in the large K limit, we find the maximum number ff c , the storage capacity, of patterns per input unit to be K ln K= ln 2 up to the leading order, which saturates the mathematical bound given by Mitchison and Durbin. Unlike current address 1 the committee machine, the storage capacity per weight remains unchanged compared with the corresponding tree-like architecture. Typeset using REVT E X 2 I. INTRODUCTION Since Gardner's pioneering work [1], statistical mechanics has been proved to be a useful tool to study feed-forward neural networks. In particular, it has allowed the derivation of the storage capacity and the generalization error of neural networks inferring a rule by examples..