What is to be assessed? Teachers’ understanding of constructs in an oral English examination in Norway

The present thesis has investigated EFL teachers’ rating orientations in an oral English examination at the upper secondary level in Norway. As part of this investigation, aspects of the teachers’ scoring behaviour, i.e. grades given, have also been studied. In addition, comparisons were made between what the teachers understand as relevant performance aspects to be tested and what the English subject curriculum and accompanying government documents define as construct-relevant. The thesis is article-based and comprises three articles and an extended abstract. The extended abstract provides a background for the investigation, a theoretical framework, a literature review, a presentation of the research design and methods used, as well as a discussion of the main findings. The articles present the three individual studies which have been conducted. A major concern throughout the thesis has been the lack of a common national rating scale in the upper secondary school context and how this may affect the validity and reliability of the scores. In Study 1 (Article 1) semi-structured interviews were used to explore 24 Norwegian EFL teachers’ general understanding of the constructs to be tested. The study found that the teachers focused on two main constructs, namely ‘communication’ and ‘content’, which in turn comprised a number of sub-constructs. Overall, the teachers understood the main constructs in the same way, but they disagreed on some of the more specific performance aspects, such as ‘pronunciation’. In addition, the study found that teachers weighted the content construct differently. The teachers working in the general studies programme put more emphasis on content than did the teachers in the vocational studies programmes. There was also evidence that some teachers focused on construct-irrelevant performance features, such as effort. Beyond this, the investigation of scoring behaviour indicated that there was fairly good agreement in the scoring of performance. Study 2 (Article 2) used semi-structured interviews and questionnaires to investigate the rating orientations of 70 EFL teachers regarding aspects of the pronunciation construct. These aspects included native speaker pronunciation and intelligibility, as well as the pronunciation of segmentals (individual sounds), word stress, sentence stress and intonation. The results showed that the teachers had widely differing views on native speaker pronunciation, but that they strongly agreed that intelligibility was important for a highscoring performance. In addition, they largely agreed that segmentals, word stress and sentence stress were important features to be assessed. As for intonation, however, the findings indicated that the teachers were either not as concerned with this feature, or unsure of its relevance. Study 3 (Article 3) employed verbal protocol analysis and semi-structured interviews to explore 10 EFL teachers’ understanding of the content construct. This construct was mainly analysed in terms of a subject matter dimension and a skills and abilities dimension. Comparisons were also made between the teachers’ perceptions of content and aspects of content identified in the subject curriculum and accompanying government documents. The results showed that the teachers had a very general understanding of subject matter, largely interpreting it in terms of ‘general world knowledge’, which may be said to correspond well with what the subject curriculum stipulates. In addition, the study found that the teachers were more concerned with the skills and abilities dimension than with the subject matter dimension, stressing the importance of higher-order thinking skills for a top-scoring performance. There was also evidence that the teachers largely had the same understanding of the construct, but that some of them disagreed on what kind of performance was indicative of the different achievement levels. These differences were largely attributed to study programme affiliation, the vocational studies teachers being more lenient in their assessment orientations than the general studies teachers. In sum, the three articles provide empirical evidence of what kind of performance aspects teacher raters attend to in a curriculum-based, oral EFL assessment context at the upper-intermediate level (Common European Framework of Reference B1/B2 level) where no common rating scale exists. Overall, the results showed that the teachers had a similar understanding of the main constructs to be tested, but that they disagreed on the more narrow performance aspects. The study also indicated that constructs such as pronunciation and content are somewhat elusive and need to be better defined in order to provide scoring outcomes that are valid and dependable. In addition, the findings suggested that the Norwegian educational authorities should consider introducing common rating scale guidelines, as well as more coherent rater training, in order to guide teachers in their assessment of oral English performance