1 Do marking reliability studies have validity?

Marking reliability studies are essential to ensure that examiners ’ marking of high stakes public assessments is appropriate, consistent and fair to candidates. The importance of such studies is confirmed by an extensive marking reliability literature which spans the early twentieth century to the present day, and covers many different forms of examination papers. There are many challenges in designing a marking reliability study, and perhaps the most significant of these is deciding whether to undertake research in a live or non-live examination context. While a live context has the greatest external and ecological validity, it is not always feasible or fair to candidates or examiners. Alternatively, using a non-live context allows for greater control of the research environment but brings into question the validity of the outcomes. For example, if participants are aware that the examination scripts have no bearing on candidates ’ real examination results, will they mark the research scripts with the same diligence as live scripts? This paper overviews the outcomes of a questionnaire distributed to 89 examiners who took part in a controlled, experimental marking reliability study. The questionnaire was designed to gauge the ecological validity of the primary (marking reliability) study by asking participants to compare the representativeness and naturalness of their behaviours during the study and under the usual ‘live ’ conditions. It is found that a small proportion of participants acknowledged that they responded differently during the study. It is suggested that these differences could represent a threat to the external and ecological validity of the study. It is concluded, however, that this is an inevitable drawback of controlled, non-live research, and that this is counter-balanced by the important insights that this and similar studies give us into the reliability of examiners ’ marking