and

∗ Equal contribution Recurrent neural networks (RNNs) are useful tools for learning nonlinear rela-tionships between time-varying inputs and outputs with complex temporal depen-dencies. Recently developed algorithms have been successful at training RNNs to perform a wide variety of tasks, but the resulting networks have been treated as black boxes – their mechanism of operation remains unknown. Here we ex-plore the hypothesis that fixed points, both stable and unstable, and the linearized dynamics around them, can reveal crucial aspects of how RNNs implement their computations. Further, we explore the utility of linearization in areas of phase-space that are not true fixed points, but merely points of very slow movement. We present a simple optimization technique that is applied to trained RNNs to find the fixed points and slow points of their dynamics. Linearization around these slow regions can be used to explore, or reverse-engineer, the behavior of the RNN. We describe the technique, illustrate it on simple examples, and finally showcase it on three high-dimensional RNN examples: a 3-bit flip-flop device, an input-dependent sine wave generator and a two-point moving average. In all cases, the mechanisms of trained networks could be inferred from the sets of fixed and slow points and the linearized dynamics around them. 2