Averaging Multiple Judges to Improve Validity: Aid to Planning Cost-Effective Clinical Research

Frequently, a researcher has the option of having each S rated on a variable by several judges rather than by a single judge. A composite judge is formed by averaging the judgments of 2 or more individual judges. Composite judges are a valuable research tool because they decrease random error of measurement and increase validity. Data analyses show that (1) a simple equation, Equation 1, predicts extremely accurately the expected validity for any given composite size; (2) standardizing each judge\u27s judgments is not necessarily important; (3) using a composite judge can yield very large validity gains when the redundancy of judges is low; and (4) there is a simple method for estimating confidence intervals around Equation 1\u27s predictions. Two of Equation 1\u27s applications in planning research (increasing the validity coefficient in a test validation study and attaining the desired level of statistical power) are discussed