Using CAT to Reduce Administration Time in 360 ° Performance Assessment

Among the challenges when collecting 360 performance assessment data are the need to (a) identify ratings that may be of questionable quality, and (b) minimize the amount of administration time. This study evaluated item response theory (IRT) appropriateness (or person-fit) indices and computer-adaptive testing (CAT) as strategies for addressing these issues. Using a sample of over 200,000 ratings collected using the the Benchmarks ® 360 ° instrument (Center for Creative Leadership, 2000), exploratory factor analyses identified a 40-item maladaptive-behaviors scale, which we calibrated using Samejima’s (1969) IRT graded response model (GRM). Results for 10,000 simulated CAT sessions showed that CAT can dramatically reduce 360 administration time, and that person-fit indices may be useful in identifying questionable response profiles. Employees in modern organizations frequently complain of being “surveyed to death. ” Wide-scale adoption of many of the technologies advocated by I-O psychologists, such as climate surveys, needs assessments, and performance appraisals, has resulted in a data collection frenzy that arguably borders on the oppressive for many organizationa