11-761 Language and Statistics Final Project Recurrent Neural Network and High-Order N-Gram Models for POS Prediction

The task of part-of-speech (POS) language modeling typically includes a very small vocabulary, which significantly differs from traditional lexicalized language modeling tasks. In this project, we propose a high-order n-gram model and a stateof-the-art recurrent neural network model, which aims at minimizing the variance in this POS language modeling task. In our experiments, we show that the recurrent neural network model outperforms the n-gram model on various datasets, and the linear interpolation of the two models, which balances the pros and cons of discriminative and generative models, has significantly reduced the perplexities. 1. Recurrent Neural Network To introduce a simple recurrent neural network language model, we first denote the input layer x, hidden layer s, word w, and output layer y. Then, the input at time t is the concatenation of word w at t and the hidden layer at previous time stamp t − 1: x(t) = w(t) + s(t − 1) The hidden layer s at time t can be represented as the function f(z) taking the sum of product input layer u and each component of the weight vector u: where the f(z) is a sigmoid function: sj(t) = f ( ∑ xi(t)uji) f(z)