Interrater Agreement Evaluation: A Latent Variable Modeling Approach

A latent variable modeling method for evaluation of interrater agreement is outlined. The procedure is useful for point and interval estimation of the degree of agreement among a given set of judges evaluating a group of targets. In addition, the approach allows one to test for identity in underlying thresholds across raters as well as to identify possibly aberrantly evaluating judges. A measure of interrater agreement is proposed, which is related to popular indexes of interrater reliability for observed variables and composite reliability. The outlined method also permits the examina-tion of underlying common sources of ratings variability, provides a useful comple-ment to the literature on interrater agreement with manifest measures, and relaxes some of its assumptions. The procedure is illustrated with numerical data. Keywords interrater agreement, latent variable modeling, reliability, raters, targets In behavioral, educational, social, and biomedical research, a study design is often-times used where several ‘‘raters’ ’ (or ‘‘judges’’)—such as psychiatrists, counselors, teachers, clinicians, evaluators, or observers—are asked to evaluate a group o