Linking the GEPT listening test to the Common European Framework of Reference

This document reports on a linking study designed to provide empirical evidence on which to base claims about the connection between the listening test suite of the General English Proficiency Test (GEPT) and the Common European Framework of Reference (CEFR). The investigation was guided by the recommended methods and procedures set out in the manual Relating language examinations to the Common European Framework of Reference for Languages: learning, teaching, assessment (Council of Europe, 2009), and entailed the stages of familiarisation, specification, standardisation and empirical validation. In addition, the project involved an innovative research design marked by a) a pilot stage to evaluate the suitability of two different standard setting methods for a linking study with listening test materials (the basket method and the modified Angoff method), and b) an asynchronous “twin-panel” approach. This methodology allowed for the inclusion and comparison of both insider and outsider perspectives through a panel of six language testing/teaching/standard setting experts without prior knowledge of the GEPT suite and a separate panel of six teachers/researchers/developers with an intimate knowledge of the GEPT suite. Preliminary alignments and cut scores were established by the researchers at the specification stage. Informed by the pilot study, a two-step adapted version of the basket method was developed which requires the standard setter to assign each item to a particular CEFR “basket” and, in addition, to decide whether a just-qualified, a mid, or a high test-taker at the particular CEFR level would already be able to answer the item correctly. After a familiarisation process with the CEFR, the GEPT, and the standard setting procedures, this modified basket method was used by the panels to judge the CEFR level of GEPT listening items. Validation checks were realised through judgement reliability analyses, judgement comparisons across panels, and qualitative analysis of recorded panel discussions. Based on the full linking study – including individual and combined panel data analyses – recommendations are made for the alignment of the GEPT listening suite with the Common European Framework of Reference. In addition, a set of suggestions is formulated in order for the GEPT listening tests to be more easily articulated with the CEFR descriptors, and in so doing to increase the validity of alignment to the framework