Evaluating aleatory uncertainty assessment

2018-08-10This dissertation aims to address two broad research questions. The first, and a fundamental one, is to what extent subjective assessments of uncertainty are informative and useful. The second, and a practical one, is what factors determine probability assessment performance. The first question is addressed in a meta-analysis designed to test whether the performance of judges in 69 different empirical studies on probability assessment is better than the performance that would be obtained when judges make uninformative guesses. The meta-analysis also provides a platform to examine the effects of various important methodological factors on probability assessment performance including the effects of expertise, elicitation methods, and assessment contexts. The results indicated that the Brier score, a performance measure of probability judgments, was statistically better than a score that would be obtained when judges made uninformative guesses although the overall quality was quite modest. Additionally, judgments from experts were significantly more calibrated and more resolute than judgments from lay individuals. In addition, aleatory uncertainty assessment resulted in a better Brier score than epistemic uncertainty assessment. Interestingly, judges performed better when they were asked to estimate probability for a single hypothesis compared to when they were asked to estimate probabilities for mutually exclusive outcomes. ❧ The second question is addressed in two forecasting studies. Particularly, these experiments were designed to evaluate alternative methods to elicit subjective probability as well as to explore the effects of individual differences on probability assessment performance. Extensive efforts were made to recruit individuals knowledgeable in assessment domains to participate in the studies. The results indicated that numeric representation of probability was more extreme but less overconfident and more resolute than verbal judgments. In addition, there were no apparent benefits of using a gamble method to elicit probability. Equally interesting, judges who received incentives based on their assessment performance did not perform better than those who did not receive such performance-based incentives. The ability to think reflectively emerged as a consistent predictor of (good) probability judgments across assessment contexts. Numeracy and active-open-minded thinking were also predictors of (good) probability judgment performance. ❧ There are three major contributions from this dissertation. First, this dissertation is the first attempt to quantify probability assessment performance from 69 primary studies on probability judgments. Second, this dissertation is one of the firsts to assess verbal aleatory uncertainty assessments. Third, this dissertation demonstrates an innovative quantification method for verbal expressions of uncertainty. These contributions provide further insights and tools for researchers to continue examining various aspects of subjective assessments of uncertainty