SCORING-RULE FEEDFORWARD AND THE ELICITATION OF SUBJECTIVE-PROBABILITY DISTRIBUTIONS

In the context of assessing subjective probability distributions, scoring rules can be used for elicitation, evaluation, and training purposes. The emphasis in the present study is on elicitation purposes and the main objective is to study the use of scoring rules for generating graphical feedforward about the consequences of a probability assessment. This notion of providing graphical feedforward about possible scores was carried through in the computerized interactive ELIcitation technique, ELI (Van Lenthe, 1993b). The effectiveness of scoring-rule feedforward was investigated in experiment A, in which 192 subjects participated. Scoring-rule feedforward, provided during the assessment process, appeared to result in a better calibration and a higher accuracy of the probability assessments. Differential effects of scoring rules were studied in experiment B, in which 304 subjects took part. Feedforward generated with different proper scoring rules (logarithmic, quadratic, spherical) yielded similar calibration and accuracy scores. A fourth condition with feedforward according an improper linear scoring rule, however, produced significantly lower scores. Finally, possible effects of the orientation (positive, negative, or both) of feedforward scores were explored, but no differences were found between conditions with mixed-score feedforward and conditions with positive-score feedforward. The ELI procedure of providing graphical scoring-rule feedforward during the elicitation process seems to be a promising method for improving the quality of probability assessments. The ELI procedure appears to be robust with respect to different types of proper scoring rules and with respect to different ranges of feedforward scores. (C) 1994 Academic Press, Inc