Age, Task Characteristics, and Acoustic Indicators of Engagement: Investigations into the Validity of a Technology-Enhanced Speaking Test for Young Language Learners

Despite calls for engaging assessments targeting young language learners (YLLs) between 8 and 13 years old, what makes assessment tasks engaging and how such task characteristics affect measurement quality have not been well studied empirically. Furthermore, there has been a dearth of validity research about technology-enhanced speaking tests for YLLs. Thus, the purpose of the current study was to explore relationships among examinee age, task characteristics, engagement, and performance in the context of gathering evidence to back a test validity argument. Following a mixed-methods approach, the investigations involved over 400 YLLs in 11 countries who responded to TOEFL Primary® Speaking test tasks (Educational Testing Service, 2013). Results from many-facet Rasch measurement revealed that, in terms of evaluation claims, tasks and raters functioned well across examinee age groups. Generalizability theory was then applied to confirm that examinees accounted for most of the score variance and that current test form configurations maximize score dependability. Fischer’s (1973, 1995) linear logistic test model results helped explain that vocabulary support, novelty, and video animation increased task difficulty, while topical choice did not. Lastly, acoustic measures of harmonicity and shimmer from the spoken responses served as indicators of engagement in a structural model showing that topical choice and novelty promoted engagement; these findings were triangulated by retrospective verbal reports from eight YLLs. Results point to the importance of including engagement in theoretical models of language performance. Also, a taxonomy of task characteristics that may support engagement is proposed to help drive a research agenda and inform test development