QUALITATIVE ANALYSIS OF TEST ITEM ATTRIBUTES

This paper presents the interim results of a set of studies undertaken to develop a much needed methodology fcr establishir.g ccnter.t validity in dcmain-referenced achievement tests. The study results are presented in the context of the larger issue of the improvement of test design. School teachers, administrators and graduate students were trained in using the Item Review Scale (IRS), and then served as item raters. The IRS provides structure for judgments of congruence between items and domain specifications. Paters were given a domain specification (elementary level geology) and eight related items to be rated fcr validity in terms of match. The items were systematically flawed along one or more cf the dimensions of the IRS. Results demonstrated high inter-rater reliability on each dimension and the instrument as a wiole. (Author/GK