Smarter Corpus-Based Syntactic Disambiguation

For years, researchers have used knowledge-intensive techniques for disambiguating during parsing. These techniques required a lot of hand-coded information, thus they would not scale to large domains. In addition, they often required the invention of pseudo-probabilities, which also do not scale, and provide ill-founded quantitative measures. The data-driven techniques, which have become popular over the past few years, seem appealing in light of this: once you have an annotated corpus, there is no need to code up knowledge bases or invent "magic numbers." However, these methods also have extensive failings, which we will detail. We present a framework for corpus-based syntactic disambiguation which pulls together the well-foundedness of the traditional approaches and the scalability of the corpus-based approaches. We also present a model of language production that places greater emphasis on lexical statistics