A Rasch measurement example in a grant application process

The purpose of this study was to investigate the effect of a Rasch-based procedure to calibrate responses for funding applications. The data set included 112 proposals and 66 readers, who independently scored randomly assigned proposals using a scoring instrument. The data were analyzed using FACETS (Linacre, 1999). The analysis indicated that the instrument succeeded in separating proposals into four distinct strata of quality. The proposal quality measures were found to be trustworthy in terms of their accuracy and stability. The 17 items on the scoring instrument were functioning as intended with a reliability of 0.98. Readers were found to be internally consistent while using the instrument to assess the quality of the proposals. Decision makers could easily translate the findings provided by the analyses and rely on the precision of the estimates to make reliable and defensible funding decisions. (Contains 5 tables and 11 references.) (Author