Using predictive differences to design experiments for model selection

Mathematical models are often used to formalize hypotheses on how a biochemical network operates. By selecting between competing models, different hypotheses can be compared. It is possible to estimate the evidence that data provides in support of one model over another. In a Bayesian framework, this is typically done by computing Bayes factors. When data is insufficiently informative to make a clear distinction, more data is required. Although the Bayesian model selection apparatus is suitable for selecting models, predicting distributions of Bayes factors is highly infeasible due to the computational complexity this involves. In this work, we propose searching for the experiment which optimally enables model selection by looking at predictive differences. We do so by simulating the posterior predictive distribution over new potential experiments. In many cases, distributions for single predictions typically show a large degree of overlap. The relations between the different prediction uncertainties depend on both the data and the model. Differences in these inter-prediction relations between competing models can be probed and used. In this work, we quantify differences in predictive distributions by means of the Jensen-Shannon divergence between predictive distributions belonging to competing models. The proposed method is evaluated by comparing its outcome to a predicted change in Bayes Factor upon simulating the experiment. Our simulations suggest that the Jensen-Shannon divergence between predictive densities is monotonically related to the increase in Bayes factor pointing toward the correct model. Therefore, it can be used to predict which experiments can effectively discriminate between models