PUB TYPE Reports Evaluative (142)

Traditional approaches to the investigation of the objectivity of ratings for constructed-response items are based on classical test theory, which is item-dependent and sample-dependent. Item response theory overcomes this drawback by decomposing item difficulties into genuine difficulties and rater severity. In so doing, objectivity of ability estimates is achieved, even though objectivity of ratings is poor. However, most item response models are too rigid to fit complexity of rater severities. Also, other types of items in the same test are excluded when estimating rater severities. These problems are addressed in this study. Several advanced models are proposed to explore severity changes over items and within items. In addition, multilevel and multidimensional models are formed to incorporate both multiple-choice items and constructed-response items in the test to increase estimating accuracy and model fit. The proposed models are made possible by a newly developed item response model, the multidimensional and multilevel random coefficients multinomial logit model. A real data set from the biology subject of the 1995 Joint College Entrance Examination in Taiwan was analyzed to demonstrate the advantages of this approach. (Contains 5 tables, 6 figures, and 35 references.) (Author) Reproductions supplied by EDRS are the best that can be made from the original document