The psychometric properties of confidence testing as a function of computerized administration, training, and development of probabilistic reasoning

The purpose of the present study was to determine whether the psychometric properties of confidence testing procedures would be enhanced through computer administration or diminished because of a lack of development of probabilistic concepts. In addition, the time required to complete a confidence test was compared to the time required to complete a traditional multiple-choice test. Three hundred and thirty-four undergraduate and graduate students participated in this study. Two hundred and eighty-six subjects completed a demographic questionnaire, the vocabulary section of Form F of the Nelson-Denny Reading test, a paper-and-pencil test of probabilistic reasoning, a training packet for confidence testing, a thirty-item vocabulary test using a confidence testing procedure, and an Attitude Toward Confidence Testing scale. Half of these subjects were given a computer administered version of the confidence test, the other half were given a paper-and-pencil version. In addition, both these groups were divided into low and high probabilistic reasoning groups using a media-split procedure. Finally, forty-eight subjects were given the thirty vocabulary items in a traditional multiple-choice format. Both time and total score were calculated for the thirty item vocabulary test. There were no significant differences in the reliability and validity coefficients for the computer administered versus paper-and-pencil conditions, or for the low versus high probabilistic reasoning conditions. There was, however, a significant increase in the time required to complete either computer administered or paper-and-pencil confidence tests when compared to a traditional multiple-choice format. Finally, students reported favorable attitudes toward the confidence testing format. In summary, it appears that computer administration does not enhance the psychometric properties of confidence tests. In addition, the time required to complete confidence tests is significantly longer than the time required to complete traditional multiple-choice tests. These results suggest that there is no psychometric benefit to computer administration and that level of probabilistic reasoning does not affect the psychometric properties of confidence testing