Approaches to the Computerized Assessment of Free Text Responses

The automated assessment of student's essays is regarded by many as the Holy Grail of computer aided assessment. Whilst a few people search for the grail, many more deny its existence. This paper describes the various approaches that have been taken over the last 40 years in an attempt to solve the problems involved with the computerized assessment of free text. The earliest approaches were founded in simple style analysis. Systems, such as Project Essay Grade (PEG), were developed upon the idea that certain surface features of an essay could be manipulated in such a way as to predict the grade that a human examiner would assign to an essay. Other methods, such as Latent Semantic Analysis (LSA), also take a statistical approach to marking, but focus on actual textual content, analyzing groupings and context. The Educational Testing Service (ETS) originally attempted to tackle the problem from a classification point of view whilst more recent work by ETS bears similarities to PEG in its statistical approach. Most of the methods currently being developed have been shown to be capable of generating essay scores that correlate with a human grader's scores at least as well as two human graders correlate with each other.. A novel approach being adopted by the authors to allow the comparison of students' essays against a model answer involves the use of theories developed for interlingual machine translation. A few different methods for knowledge representation, and their current uses in machine translation are presented. Panlingua, an idea for knowledge representation developed by Chaumont Devin, is based on semantic network research. Using a system of nodes over four layers it attempts to model how the brain might translate from sensory patterns it sees or hears at the top l..