Dynamics of learning near singularities in layered networks

We explicitly analyze the trajectories of learning near singularities in hierar-chical networks, such as multilayer perceptrons and radial basis function networks which include permutation symmetry of hidden nodes, and show their general prop-erties. Such symmetry induces singularities in their parameter space, where the Fisher information matrix degenerates and strange learning behaviors, especially the existence of plateaus in gradient descent learning, arise due to the geometric structure of singularity. We plot dynamic vector fields to demonstrate the univer-sal trajectories of learning near singularities. The singularity induces two types of plateaus, the on-singularity plateau and the near-singularity plateau, depend-ing on the stability of the singularity and the initial parameters of learning. The results presented in this paper are universally applicable to a wide class of hier-archical models. Detailed stability analysis of the dynamics of learning in radial basis function networks and multilayer perceptrons will be presented in separate papers.