Modeling the effect of item order on the test score

The question whether order of the items in the test has an effect on the test score already has been investigated in different fields, such as personality and clinical psychology, survey research, and educational measurement. Studies on the item-order effect in different contexts showed opposite results. For example, in educational measurement, both an increased performance (or so-called practice effect) and a decreased performance (or a so-called fatigue effect) towards the end of the test have been reported (Bejar, 1985; Kingston & Dorans, 1982). A more intriguing example of opposite results is related to effect of item order on the relationship of the item with the latent trait. Findings in personality assessment suggest that the number of items preceding the item may increase the correlation with the latent trait (e.g., Hamilton & Shuminsky, 1990; Knowles, 1988). In educational measurement on the other hand, a decrease in the correlation of the end-of-test items with the latent trait can be expected in some situations given that test takers may change to a pure guessing strategy (e.g., Yamamoto & Everson, 1997). Hence, as Knowles (1988) concluded, studies on the item-order effect show that measuring may change the measure. In the past, several models have been proposed to investigate the effect of item order on the test scores. These models belong to the family of item response theory (IRT) and structural equation models (SEM) The proposed models differ along the following dimensions: (a) whether the effect is investigated in a design of test administration where different item orders are presented or not; (b) whether the effect is investigated at the level of the test score versus at the level of individual items and (c) whether individual differences in the effect of item position are taken into account or not. In the present paper, a new model is proposed that analyses the item-order effect at the level of items but that takes individual differences into account as well. The model belongs to the family of IRT models and combines the linear logistic test model (Fischer, 1973) with random-effects at the person side. After the presentation of the model, an application is described investigating item-order effects on a computer-based test on listening comprehension of French as a foreign language.status: publishe