Modeling item position effects within an IRT framework: A slice of PISA

In educational and psychological measurement, it is assumed that item responses and test scores depend only on the attribute of interest, and that the measurement is invariant with respect to administration conditions. However, it has been shown that context effects might violate this measurement invariance. The present presentation focuses on the impact of the position of an item on item characteristics, and uses an IRT framework to detect and model these effects. As an application, the position effect on item difficulty in the data from the reading assessment from the PISA 2009 study will be explored (in collaboration with Johannes Hartig, Janine Buchholz and Rianne Janssen). As in most large scale assessments, the PISA study uses alternate test forms with different item orders. Within the PISA rotated block design, item position and item content are varied independently, which allows an analysis of the effect of item position. Aside a main effect of item position on item difficulty, an associated random effect is included in the model, to capture individual differences in item position effects, i.e. students' persistence during the assessment. Furthermore, the modeling approach allows decomposing the variance in the position effect into within- and between-school effects. The model allows an estimation of (1) the general effect of the item position, (2) the amount of variance in persistence within and between schools, and (3) the correlation between the performance level and the persistence within and between schools. Responses to 95 reading test items were used in the analysis. All effects of interest were analyzed separately for each country, allowing a comparison of the effects across countries. A significant general effect of item position was found, items performance decreases with item position in every country. Significant individual differences were found within and between schools; however, the variance in students’ persistence is smaller for countries with a higher international performance level. The pattern of correlations between students’ performance and persistence varies across countries, positive as well as negative correlations are found, within and between schools.status: publishe