Sequence transduction with recurrent neural networks

Many machine learning tasks can be ex-pressed as the transformation—or transduc-tion—of input sequences into output se-quences: speech recognition, machine trans-lation, protein secondary structure predic-tion and text-to-speech to name but a few. One of the key challenges in sequence trans-duction is learning to represent both the in-put and output sequences in a way that is invariant to sequential distortions such as shrinking, stretching and translating. Recur-rent neural networks (RNNs) are a power-ful sequence learning architecture that has proven capable of learning such representa-tions. However RNNs traditionally require a pre-defined alignment between the input and output sequences to perform transduc-tion. This is a severe limitation since finding the alignment is the most difficult aspect of many sequence transduction problems. In-deed, even determining the length of the out-put sequence is often challenging. This pa-per introduces an end-to-end, probabilistic sequence transduction system, based entirely on RNNs, that is in principle able to trans-form any input sequence into any finite, dis-crete output sequence. Experimental results for phoneme recognition are provided on the TIMIT speech corpus. 1