A comparison of aggregation methods of subjective probability distributions

One of the goals of psychological research on subjective judgments is to develop procedures that can improve judgment aggregation quality. The need to aggregate various judgments arises since in many cases the decision maker is uncertain about the possible outcomes of his decisions solicits suggestions from multiple advisors. In this paper, we study the quality of aggregation of multiple subjective probability distributions of future temperatures, using data collected by Abbas, Budescu, Yu and Haggerty (2008), as a function of 4 factors ??? the elicitation method (Fixed Probability versus Fixed Variable), the aggregation method (combining directly points on the distribution or aggregating parameters of fitted distributions), the aggregation statistic (using the mean or the, more robust, median to represent the aggregated values), and group size (we used data from 32 judges and we compare results of 200 replications of sub-groups of increasing size: the 32 single judges (n=1), 16 pairs of judges (n=2), 8 groups of n=4 judges, 4 groups of n=8 judges, 2 groups of n=16 judges, and a summary of all n=32 judges). The quality of aggregation is measured primarily by the closeness of the estimated probability distribution to the reference distribution based on historical data. We observed that as group sizes increases, aggregation quality improves (closer fit to the historical values) and it matters less which judges are aggregated and how the judgments are aggregated. Aggregates based on FP assessment generate higher quality than aggregates based on FV assessment under most circumstances. When FP is adopted, point aggregation generates better results than parameter aggregation. If FV has to be adopted for practical reasons, using parameter aggregation with mean may produce higher quality results