The Influence of Direct Versus Indirect Observation, Candidate Report Format, and Assessor Training on the Accuracy of Assessor Ratings

Within the assessment center method, assessors\u27 preliminary evaluations of candidates\u27 performance are typically based on information obtained through direct observation of the candidate\u27s performance in the situational exercise, or on another assessor\u27s report of that performance. This variation is somewhat disconcerting, however, in that its impact on assessor ratings remains largely unstudied. The primary focus of the present study was to assess the differential effects of observation type (i.e., direct observation, dimension-specific report, narrative report) on various measures of rating accuracy. In addition, the present study investigated cognitive modeling as an assessor training strategy, and its impact on rating accuracy. Seventy-three undergraduates majoring in business administration were either trained or not trained, and either observed and rated nine videotapes depicting individuals conducting performance review sessions, or reviewed and evaluated corresponding dimension-specific or narrative reports describing the same performance. A two (cognitive modeling training, no training) by three (direct observation, dimension-specific report, narrative report) analysis of variance design was used to assess the effects of training and observation type on rating accuracy. Moderate support was found for the hypotheses that direct observation would yield greater accuracy than report-based ratings. While ratings obtained in the direct observation condition were generally more accurate than narrative-report based ratings, dimension-specific report ratings produced superior accuracy. In addition, those participants receiving the cognitive modeling training, as predicted, were significantly more accurate in their ratings than the no-training participants. These results suggest that in assessment situations where assessors must rely on reports to evaluate candidate performance, those reports should be constructed in a dimension-specific format. Furthermore, cognitive modeling appears to be a viable strategy for assessor training. Future research should examine reports which more closely typify actual assessment center products and their effects on rating accuracy. The cognitive modeling approach to assessor training should also be given greater attention in further study