Statistical and psychometric issues in the measurement of educational achievement trends: Examples from the national assessment of educational progress

Like all studies involved in the assessment of trends in educational perfor-mance, the National Assessment of Educational Progress (NAEP) is con-fronted with an array of unresolved methodological and philosophical issues. One of the basic dilemmas faced by NAEP is how to measure performance change while remaining responsive to advances in curriculum and the tech-nology of assessment. NAEP has become much more cautious about making seemingly insubstantial changes in the assessment because of the so-called NAEP reading anomaly—an apparently steep drop between 1984 and 1986 in estimated reading proficiency that was found to have resulted in part from changes in the order and context in which items appeared. Other issues that NAEP must consider in reporting performance trends are the effect of measurement scale indeterminacies and the ways in which interpretation of trend results can depend on the statistics that are selected for comparing proficiency distributions over time. The primary goal of NAEP is to measure changes in educational perfor-mance over time. The purpose of this article is to explore some of the unresolved methodological and philosophical issues involved in the assess-ment of performance trends. Consideration of these issues may suggest new methodologies and new ways of displaying and interpreting results. The Much of this article was inspired by discussions with NAEP's research staff and its Design and Analysis Committee. It has benefited from reviews by John B