A cautionary note on estimating the reliability of a mastery test with the beta-binomial model

Based on recently published papers, it might be tempting to routinely apply the beta-binomial model to obtain a single administration estimate of the reliability of a mastery test. Using real data, the paper illustrates two practical problems with esti-mating reliability in this manner. The first is that the model might give a poor fit to data, which can seriously affect the reliability estimate, and the sec-ond is that inadmissible estimates of the param-eters in the beta-binomial model might be ob-tained. Two possible solutions are described and il-lustrated. In recent years, efforts have been directed toward deriving ways of studying and characterizing mastery and criterion-referenced tests. A summary of the statistical and psychometric techniques that have evolved can be found in the 1980 special issue of Applied Psychological Measurement (see, also