PUB TYPE Reports Evaluative (142) Speeches/Meeting Papers (150)

It is erroneous to extend or generalize the inter-rater reliability coefficient estimated from only a (small) proportion of the sample to the rest of the sample data where only one rater is used for scoring, although such generalization is often made implicitly in practice. It is shown that if inter-rater reliability estimate from part of a sample is available, the score reliability for the rest of the sample data rated by only one rater can be estimated both within the classical reliability theory framework, and within the framework of generalizability theory. As intuitively expected, score reliability for the data for which only one rater is used for scoring is always lower than the score reliability for the portion of sample data for which two raters are used. A sample of published studies is provided from difference disciplines that gives inter-rater reliability coefficients obtained from a small proportion of a sample. For this sample of published studies, by applying the method discussed in this paper, the estimated score reliability is given for the data rated by only one rater. (Contains 1 table and 20 references.) (Author/SLD) * Reproductions supplied by EDRS are the best that can be made * * from the original document. * RATER.V