An exploration of examinee abilities, rater performance, and task differences using diverse analytic techniques

Thesis (Ph.D.)--University of Hawaii at Manoa, 2008.The results of the current study indicated that: (a) the two group oral tasks and three semi-direct speaking tasks are not comparable in difficulty, (b) the method effect of the group oral tasks is so strong as to attenuate trait effect, (c) the three tasks demonstrate a strength in eliciting different aspects of oral proficiency such as accuracy, complexity, and lexical diversity, and (d) there is no statistically meaningful association between how the examinees perceived difficulty of the tasks and how they performed on them. In concluding this study by summarizing the findings, pedagogical implications, limitations, and future research avenues are also discussed.This dissertation aims to explore the extent to which three test tasks---discussion, information gap, and semi-direct---are comparable in L2 speaking assessment. One hundred twenty two students in a Japanese university performed the three speaking test tasks in English, and their performance rated by ten judges was analyzed using diverse statistical analytic techniques such as the Multi-facet Rasch Method, ANOVA, multitrait-multimethod confirmatory factor analysis, and speech sample analysis. Using ability scores, the comparability of the three test tasks were examined, especially, concerning examinee abilities, task differences, and rater performance.While the literature is rich with respect to the use of tasks for L2 classroom practices (e.g., Doughty & Pica, 1986; Pica, 2005; Pica, Kang, & Sauro, 2006) and task features and their impact on L2 examinee performance (e.g., Brown, Hudson, Norris, & Bonk, 2002; Bygate, Skehan, & Swain, 2001; Long, 1985, 1989, 2005; Long & Norris, 2000; Norris, Brown, Hudson, & Yoshioka, 1998; Robinson, 1995, 1998, 2001a, 2001b; Skehan, 1996; Skehan, 1998; Skehan & Foster, 1997, 1999, 2001), researchers in L2 assessment have just begun to explore the use of speaking tasks and their implications to examinee performance (e.g., Elder, Iwashita, & McNamara, 2002; Fulcher, 1996; Iwashita, McNamara, & Elder, 2001; Van Moere, 2006). In particular, studies are not common that examined the extent to which different speaking tasks are comparable in assessing L2 oral proficiency.Includes bibliographical references (leaves xxx-xxx).Also available by subscription via World Wide Web251 leaves, bound 29 c