Criterion-referenced measurement: A Bayesian validation of the beta-binomial prediction model.

This paper examines the use of the beta-binomial distribution to predict true mastery following a criterion-referenced test. It has been observed in prior research that despite a well selected cutting score for pass/fail classification, performance above the score does not insure true mastery. Earlier studies regarding the Graduate Record Examination, The Armed Services Vocational Aptitude Battery, physician licensure, and nursing licensure indicate concern over the predictive qualities of criterion-referenced testing. In higher education, researchers substantiate changes in the concept of validity, and its migration from construct to context emphasis in order to assess the predictive qualities of criterion-referenced testing. While the beta-binomial distribution is the statistical description of choice for determining the probability that mastery has taken place, this study employs extensive field data to revise these probabilities using a decision-theoretic, Bayesian method. In so doing it is hypothesized that Bayesian statistics can act as a validation mechanism for the a priori, beta-binomial predictions. The use of general aviation mishap data not only facilitated a confirmation of the hypothesis but also resulted in recommendations for improving the criterion-referenced processes of pilot certification. When generalized to issues in higher education, this study suggests that decision-theoretic methods can be used advantageously in research. Moreover, the techniques employed here offer alternatives to classical correlation studies in higher education.PhDEducationUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/104322/1/9513388.pdfDescription of 9513388.pdf : Restricted to UM users only