Operational validation of health economic decision analytic models

Objectives: To validate health economic (HE) models by means of statistical comparison of model outcomes against empirical observations. Such a comparison is structured and the applicability of several existing validation techniques is discussed, with a special focus on statistical testing. When standard methods (95%-confidence intervals) are used several problems, both of a technical and philosophical nature, are encountered. These problems are discussed. A new statistical approach is consequently proposed. Methods: The proposed method can be applied to validate HE models when the uncertainty around the input parameters of the model is assessed via probabilistic sensitivity analysis (PSA). It is based on the idea of establishing a level of accuracy in advance for the empirical observations and model outcomes should meet. If the model result falls within the limits determined by the pre-required accuracy, then the model result is considered valid. The number of valid results obtained in a PSA defines a measure of the reliability of the model. Embodying the method in a Bayesian framework allows defining such a reliability measure with statistical properties. Results: Existing approaches suffer from technical and interpretational problems. In addition, these methods are lacking a measure of overall reliability. Our new method (1) departs from classical statistical techniques, circumventing the noted problems, (2) can be used for both cohort and patient-level models and (3) makes use of all PSA outcomes. The method is demonstrated with the help of a case study in a published diabetes model (MICADO). Conclusions: Standard statistical techniques have to be applied very carefully on the comparison of model outcomes to empirical observations. They suffer from several problems. A new promising Bayesian approach is proposed that solves some of these issues. Our new method allows stepwise validation of the model as new data becomes available, which may increase the model's validation status