Dynamical Singularities in Online Learning of Recurrent Neural Networks

Abstract — We numerically and theoretically demonstrate var-ious singularities, as a dynamical system, of a simple online learning system of a recurrent neural network (RNN) where RNN performs the one-step prediction of a time series generated by a one-dimensional map. More specifically, we show first through numerical simulations that the learning system exhibits singular behaviors (“neutral behaviors”) different from ordinary chaos, such as almost zero finite-time Lyapunov exponents, as well as inaccessibility and power-law decay of the distribution of learning times (transient times). Also, we show through linear stability analysis that, as a dynamical system, the learning system is represented by a singular map whose Jacobian matrix has eigenvalue unity in the whole phase space. In particular, we state that the singularity as a dynamical system (shown by the second method) provides a basic reason for the neutral behaviors (shown by the first method) exhibited by the learning system. I