Low-rank matrix factorization for deep neural network training with high-dimensional output targets

While Deep Neural Networks (DNNs) have achieved tremen-dous success for large vocabulary continuous speech recognition (LVCSR) tasks, training of these networks is slow. One reason is that DNNs are trained with a large number of training parameters (i.e., 10-50 million). Because networks are trained with a large number of output targets to achieve good performance, the majority of these parameters are in the final weight layer. In this paper, we propose a low-rank matrix factorization of the final weight layer. We apply this low-rank technique to DNNs for both acoustic modeling and lan-guage modeling. We show on three different LVCSR tasks ranging between 50-400 hrs, that a low-rank factorization reduces the num-ber of parameters of the network by 30-50%. This results in roughly an equivalent reduction in training time, without a significant loss in final recognition accuracy, compared to a full-rank representation. Index Terms—Deep Neural Networks, Speech Recognition 1