Teacher Evaluation of Item Formats for an English Language Proficiency Assessment

An abstract of the thesis of José Luis Perea-Hernández for the Master of Arts in Teaching English to Speakers of Other Languages presented February 11, 2010. Title: Teacher Evaluation of Item Formats for an English Language Proficiency Assessment. Language testing plays a significant role in the collection of systematic information about English Language Learners\u27 ability and progress in K-12 public schools in Oregon. Therefore, the following thesis reports on a process of the development of this assessment as I was at some point one of the members in the test development team and then drew upon teachers\u27 opinions and judgments to compensate for a gap in this process. For instance, despite the fact that Oregon item-writers produced test items in summer 2004, they identified some deficiencies as they followed the process. Therefore, the need of more examples of item formats in order to develop better items, a proper identification of materials for reading and listening and mainly the lack of detailed item test specifications was imminent. Unfortunately, item writers did not posses a guiding document that would allow them to generate items. The item writing process was not `spec-driven\u27. Hence, these drawbacks in this test (larger project), provide the basis and rationale for this thesis. Thus, the aim of the present study was to identify what item formats worked best based on professional teacher judgment. Then, once identified these were used to detect what items could potentially be reverse-engineered into specifications for future studies or developmental stages of this assessment. Therefore, the purpose of this thesis was to analyze through professional teacher-judgment the suitability of item formats, test items and the construct in a way that would be helpful for future item-writing sessions, but most importantly to the better understanding of teachers devising items for this test. Teachers determined what items and item formats were more suitable in a testing situation for the diverse English language learners according to their relationship to the construct of the test, practicality, cognitive level, appropriateness, and similar tasks as used in classroom instruction. Finally, the outcomes based on teacher feedback allowed the required data to establish a process to reverse-engineer test items and item formats into item test specifications. Thus, based on this feedback a more systematic process was illustrated to demonstrate how item formats can be the basis to reverse-engineer item test specifications