Comparing in-person and online modes of expert elicitation

Expert elicitation, a method of developing probability distributions over unknown parameters, traditionally involves in-person interviews by a trained analyst. There is growing interest in using the internet to enable participation of larger, more distributed groups of experts. However, analysts have questioned the quality of judgements elicited online rather than in person. We systematically compare online and in-person elicitation modes, finding no significant difference between the two modes across multiple measures: the two modes are similar in accuracy, uncertainty ranges, number of surprises, fatigue, and the substance of qualitative comments. These findings have an important caveat: many elicitation questions were subject to problems in online administration that made it impossible to compare to in-person results. We conclude that, although online elicitations represent a less resource-intensive option for large expert elicitations, they may require a higher level of testing and quality control since there is no analyst to catch errors or clarify small misunderstandings