IDENTIFIERS *National Assessment of Educational Progress; *Standard Setting

This paper describes the feedback included in operational achievement level setting (ALS) procedures for the National Assessment of Educational Progress (NAEP). It does not describe the feedback form field trials, pilot studies, or other research studies related to the NAEP. The NAEP ALS process includes three rounds of item-by-item ratings, and feedback is provided after each round. This means that panelists have additional information to consider for subsequent rounds of ratings, but additional training is required for no more than one of two new forms of feedback for each round. Most of the feedback is aimed at increasing interjudge and intrajudge consistency. The types of feedback provided are: (1) cutscores and standard deviations; (2) student performance data; (3) interrater consistency data; (4) whole booklet data; (5) interrater consistency data, including "Reckase " charts of performance data; and (6) consequences data. The use of each type of feedback is discussed, and panelists ' evaluations of the feedback provided are noted. Panelists ' responses indicate high levels of understanding about and confidence in using the feedback. The Reckase charts appear to be a significant addition to the array of feedback information. (Contains 9 tables, 17 figures, and 22 references.) (SLD) Reproductions supplied by EDRS are the best that can be made from the original document