Automated Scoring of Speaking Items in an Assessment for Teachers of English as a Foreign Language

This paper describes an end-to-end proto-type system for automated scoring of spo-ken responses in a novel assessment for teachers of English as a Foreign Language who are not native speakers of English. The 21 speaking items contained in the as-sessment elicit both restricted and moder-ately restricted responses, and their aim is to assess the essential speaking skills that English teachers need in order to be effec-tive communicators in their classrooms. Our system consists of a state-of-the-art automatic speech recognizer; multiple fea-ture generation modules addressing di-verse aspects of speaking proficiency, such as fluency, pronunciation, prosody, gram-matical accuracy, and content accuracy; a filter that identifies and flags problematic responses; and linear regression models that predict response scores based on sub-sets of the features. The automated speech scoring system was trained and evaluated on a data set involving about 1,400 test takers, and achieved a speaker-level cor-relation (when scores for all 21 responses of a speaker are aggregated) with human expert scores of 0.73.