1 Running head: Invariance in Rater-mediated Assessments Approximate Measurement Invariance in Cross-classified Rater-mediated Assessments

An increasingly common high-stakes form of rater-mediated assessment in education is classroom observations of teaching quality. An important assumption underlying meaningful comparisons of scores in these and other types of rater-mediated assessments is that measurement is commensurate or invariant across raters. However, research has shown that there are often important differences in raters ’ judgments that potentially place scores from different raters on fundamentally different scales. Despite evidence of rater differences, scores from different raters are routinely treated as if they were exchangeable and are often used to make high-stakes comparative decisions. In this study, we developed a method to accommodate measurement noninvariance across raters when measurements are cross-classified within two distinct hierarchical units. We conceptualized a random item effects cross-classified graded response model and used random discrimination and threshold effects to test, calibrate, and account for measurement noninvariance among raters. By leveraging empirical estimates of rater-specific deviations in the discrimination and threshold parameters, the proposed method helps to identify noninvariant items, and empirically estimate and adjust for this noninvariance within a cross-classified framework. Within the context of teaching evaluations, the results of a case study suggested substantial noninvariance across raters and that establishing an approximately invariant scale through random item effects improves model fit and predictive validity. 3 An increasingly common high-stakes form of rater-mediated assessment in education i