Natural Gradient Matrix Momentum

Natural gradient learning is an efficient and principled method for improving on-line learning. In practical applications there will be an increased cost required in estimat-ing and inverting the Fisher information ma-trix. We propose to use the matrix momen-tum algorithm in order to carry out efficient inversion and study the efficacy of a sin-gle step estimation of the Fisher information matrix. We analyse the proposed algorithms in a two-layer neural network, using a statis-tical mechanics framework which allows us to describe analytically the learning dynam-ics, and compare performance with true nat-ural gradient learning and standard gradi-ent descent.