Issues in the Interpretation of PISA in Ireland: A Study of the Content and Design of PISA with Comparative Analyses of the Junior Certificate Examinations and TIMSS (1 Volume)

The results of the Programme for International Student Assessment (PISA) have been the subject of attention in media reports and ministerial speeches in Ireland since 2001. Yet, to date, there has been no examination of the appropriateness of the data it yields to inform educational policy. Analyses described in this dissertation were carried out to identify aspects of the design and interpretation of the PISA assessments of reading and mathematics which may be problematic and/or at odds with conclusions drawn. Three major questions are addressed. First, what does PISA tell us about achievements of students in Ireland? While PISA is intended to be used for educational improvement, the fact that it does not purport to assess school-based knowledge and skills could be problematic. In this context, the extent to which the PISA achievement measures are similar to, or differ from, the national curriculum (Junior Certificate syllabus and examinations) is examined. The PISA reading measure was found, by and large, to be compatible with Junior Certificate English. However, the diverging philosophies underlying PISA and Junior Certificate mathematics result in notable disparities between the two assessments which pose challenges in interpreting the results (although these disparities may serve a potentially valuable ‘enlightenment’ function and act as a trigger for curriculum review). Both PISA and the Junior Certificate Examination were found to be of very limited utility in describing performance at the low end of the achievement distribution. Analyses of non-response in the Irish datasets for PISA 2000 and 2003 reveal significant bias arising from student (but not school) non-response. This finding renders claims made by the OECD and Irish media about the characteristics of low achievers problematic and strengthens the argument that PISA is not well suited to describing characteristics of low achievers in Ireland. The second question considered is what PISA tells us about the equity of achievement outcomes in Ireland. Analyses of the between-school variance statistic were used to address this question since is widely cited as an indicator of educational equity. However, when inferences are being made on the basis of between-school variance, no cognizance is taken of how sample design and other characteristics of PISA might have impacted on its magnitude. This issue is of particular relevance in Ireland where studies have revealed large achievement differences associated with class allocation within schools. Furthermore, the nature of the achievement measure used (in particular, its curriculum sensitivity) would also appear relevant to the interpretation of the significance of between-school variance. Comparative analyses of the TIMSS 1995 and PISA data indicated that between-'school' variance in Ireland is much larger when the sample design involves the selection of intact classes (in TIMSS) rather than on the basis of students' age (in PISA). Furthermore, school-dependent and curriculum-sensitive measures tend to be associated with higher between-school variance. The third question relates to what PISA tells us about the determinants of the achievements of Irish students. This question is of some significance since the impact of schools’ social intake on student achievement, and the extent to which school practice variables explain achievement, are given prominence in the results of PISA and in other surveys both nationally and internationally. A comparison of multilevel models of Irish student achievement on PISA 2000, PISA 2003 and TIMSS 1995 using achievement on the international tests and the Junior Certificate Examinations suggests that the impact of social intake is larger when surveys use a sample design based on intact classes, while school-dependent and curriculum-sensitive measures may be more sensitive to school practice variables. In the conclusions, some policy implications are described, the limitations which these findings place on the interpretation of results considered, some improvements to the design of PISA that may overcome some of the limitations suggested, and areas for future research proposed