A Clockwork RNN

Sequence prediction and classification are ubiqui-tous and challenging problems in machine learn-ing that can require identifying complex depen-dencies between temporally distant inputs. Recur-rent Neural Networks (RNNs) have the ability, in theory, to cope with these temporal dependencies by virtue of the short-term memory implemented by their recurrent (feedback) connections. How-ever, in practice they are difficult to train success-fully when long-term memory is required. This paper introduces a simple, yet powerful modifica-tion to the simple RNN (SRN) architecture, the Clockwork RNN (CW-RNN), in which the hidden layer is partitioned into separate modules, each processing inputs at its own temporal granularity, making computations only at its prescribed clock rate. Rather than making the standard RNN mod-els more complex, CW-RNN reduces the number of SRN parameters, improves the performance significantly in the tasks tested, and speeds up the network evaluation. The network is demonstrated in preliminary experiments involving three tasks: audio signal generation, TIMIT spoken word clas-sification, where it outperforms both SRN and LSTM networks, and online handwriting recogni-tion, where it outperforms SRNs