Combining computational models of short essay grading for conceptual physics problems

The difficulties of grading essays with natural language processing tools are addressed. The present project investigated the effectiveness of combining multiple measures of text similarity to grade essays on conceptual physics problems. Latent semantic analysis (LSA) and a new text similarity metric called Union of Word Neighbors (UWN) were used with other measures to predict expert grades. It appears that the best strategy for grading essays is to use student derived ideal answers and statistical models that accommodate inferences. LSA and the UWN gave near equivalent performance in predicting expert grades when student derived ideal answers served as a comparison for student answers. However, if ideal expert answers are used, explicit symbolic models involving word matching are more suitable to predict expert grades. This study identified some computational constraints on models of natural language processing in intelligent tutoring systems. © Springer-Verlag 2004