Dynamics of unconstrained networks are similar to low-rank RNNs.

A) We also trained RNNs without rank constraint. For these networks initial entries in the recurrent weight matrix J were drawn from a zero-mean Gaussian with variance . Here, we add a regularisation term to the loss, which keeps the average firing rates close to 0, to avoid a rate-coding solution (Eq 6). We set g to 0.6 and took a learning rate of 0.001, with the training setup otherwise as for the low-rank networks (Training). In order to find a basis similar to the one used for plotting the dynamics of the low-rank RNN we took the following approach. First we calculated basis vectors for the activity due to recurrent dynamics: J tanh(x(t)), by performing a Principal Component Analysis (singular vector decomposition): , where X is an matrix containing the activity of all units for one period of oscillation of each type of trial (stimulus a and b). We took the first two columns (principal components), u1 and u2, of U, as well as the input vector I(osc) orthogonalised with respect to these two principal components as basis for X. This basis retains 77% of the variance of X (measured by r2). Since now, similar to the low-rank case, we can write the projection of x(t) on as a function of θ, we can plot trajectories with coordinates (θ, , and we obtain two stable cycles, linked in phase-space, as in Fig 2. B) Here we train full-rank networks without regularisation, and find dynamics lying on two non-linked cycles, similar to the rate-coding model in Fig 2. The basis is constructed similar to A), here explaining 87% of the variance. (TIF)