Evaluation of Natural Language Processors

253 p.Thesis (Ph.D.)--University of Illinois at Urbana-Champaign, 1981.Despite a large amount of research on developing natural language understanding programs, little work has been done on evaluating their performance or potential. The evaluations that have been done have been unsystematic and incomplete. This has lead to uncertainty and confusion over the accomplishments of natural language processing research.The lack of evaluation can be primarily attributed to the difficulty of the problem. The desired behavior of natural language processors has not been clearly specified. Partial progress toward the eventual goals for natural language processors has not been delineated, much less measured.This thesis attempts to clarify some of the difficulties behind evaluating the performance of natural language processors. It also proposes an evaluation method that is designed to be systematic and thorough. The method relies on considering a natural language processor from three viewpoints in the light of several taxonomies of issues relevant to natural language processing. Finally, an evaluation is described of PLANES, a natural language database query system.U of I OnlyRestricted to the U of I community idenfinitely during batch ingest of legacy ETD