Using an effective Boltzmann machine to learn context dependencies of a sequence

It has been recognized that current Artificial Neural Network (ANN) systems that employ windowing techniques to learn context dependencies in sequences have many deficiencies. One variant of this type of system that attempts to overcome many of these deficiencies is the Effective Boltzmann Machine (EBM). The EBM which is based on the Boltzmann Machine (BM) has the ability to perform completion and to provide an energy measure for the solution. In this paper, we extend the EBM and show that the system has many desirable properties. This paper reports two major improvements to the EBM. First, where as in the past, a BM is used to learn the local contexts of a sequence, we show that the EBM itself is an architecture suitable for learning local contexts. Our initial experiments show that it outperforms a window based BM. Second, we demonstrate that, with this new training scheme, a multilayer EBM can be constructed to extend the effective window size. This means that learning contextual dependencies is not limited by the window size. This is done by utilizing the state information of the first EBM hidden layer to train that of the second EBM layer. The effect of this long range effective window is demonstrated by experiments