Probabilistic Predictive Elicitation

Unlike the traditional machine learning approaches that rely solely on data, Bayesian machine learning models can utilize prior knowledge on the data generating process, for instance in form of information about plausible outcomes. More importantly, Bayesian machine learning models use the prior information as the base knowledge, on top of which the learning from observations is built on. The process of forming the prior distribution based on subjective probabilities is called prior elicitation, and that is the focus of this thesis. Although previous research has produced methods for prior elicitation, there has not been a general-purpose solution. Particularly, the methods introduced previously have focused on specific models. This has limited the applicability of prior elicitation, and in some cases, required the expert to have a deep understanding of different aspects of the Bayesian modelling. Additionally, the more general predictive elicitation methods in previous research have not accounted for the uncertainty regarding experts' judgements. This is important, since even the most accurate elicitation methods cannot remove all imprecision in expert judgements. Because of these reasons, prior elicitation has remained somewhat underrated and underused in the modern Bayesian workflow. This thesis provides a theoretical basis and validation of a novel prior elicitation method, which was first introduced by Hartmann et al. Particularly, this principled statistical framework called probabilistic predictive elicitation 1) makes prior elicitation independent on the specific structure of the probabilistic model, 2) handles complex models with many parameters and potentially multivariate priors, 3) fully accounts for uncertainty in experts' probabilistic judgements on the data, and 4) provides a formal quality measure indicating if the chosen predictive model is able to reproduce experts' probabilistic judgements. We extend the published work in multiple ways. First, we provide more thorough literature reviews on different prior elicitation approaches as well as on methods for the expert elicitation. Second, we continue the discussion about technicalities, implementation and applications of the proposed methodology. Third, we report two unpublished experiments using the proposed methodology. In addition, we discuss the methodology in the context of the modern Bayesian workflow