Weight Space Learning Trajectory Visualization

Visualizing the trajectory followed through weight space when a feed-forward neural network is trained is made difficult by the very large dimensionality of the weight space in networks of practical size. A new approach, using Principal Component Analysis, is shown to be effective in a realistic learning scenario at capturing the information contained in the learning trajectory (or a section of it) so that it can be visualized. The results of performing Principal Component Analysis on weight space learning trajectories are further explored in relation to the dynamics of the learning process, the amount of variance captured by a subset of principal components, and the direction in which the learning trajectory evolves. 1. Introduction Learning in a feed-forward neural network is often viewed as a search for a minimum of a multidimensional cost (or error) function. The weights in a network (including the bias) are the only parameters adjustable during training. This means that the dimen..