Constructing Novel Recurrent Neural Network Architectures Using Hamiltonian Dynamics

Recurrent neural networks (RNNs) have gained a great deal of attention in solving sequential learning problems. The learning of long-term dependencies, however, is still an open problem caused by the vanishing or exploding gradient. This problem corresponds to the loss of information in the sequential learning data. The goal is therefore to construct RNNs which can preserve the information at least for very long times. In this thesis we will provide the connection between RNNs and Hamiltonian dynamics which is the natural tool in theoretical physics when it comes to preservation characteristics. We will construct an RNN based on a specific Hamiltonian system and call it Hamiltonian recurrent neural network (HRNN). We will derive a sensitivity analysis for the HRNN and show that we can directly control the hidden states gradient which is causing the vanishing or exploding gradient problem. We demonstrate the superiority of the HRNN compared to other state-of-the-art RNNs by providing several benchmark experiments. Finally, we present a real-world application for the HRNN by forecasting the future trends of a stock based on historical financial data