Statistical analysis of reliability-validity studies

Reliability and validity studies are common in epidemiology, especially in the course of the development of measure instruments. Reliability is typically assessed quantitatively, through an appropriate coefficient, such as the Intra-class Correlation Coefficient (ICC). Validity, on the other hand, is assessed more informally through a series of tests and checks: for instance, construct validity may be established by testing the significance of factors that are supposed to influence the measure at hand. In general an ICC is calculated as a ratio of variance components to the total variance. Therefore the first step in the calculation of an ICC is the estimation of variance components from an appropriate statistical model. This thesis presents two approaches to the estimation of variance components in the context of reliability and validity studies: One is the ANOVA approach, based on the method of moments and valid especially for the case of balanced data. The other is the mixed linear model approach, for the more general case of unbalanced data. Furthermore, a general framework is developed which permits treatment of reliability and validity within the same statistical model, By means of this statistical model, a special case of the mixed linear model, appropriate ICCs for both reliability and validity can be computed, while construct validity can be established by testing the significance of appropriate fixed effects. The delta method and the bootstrap are proposed for the calculation of the standard errors and confidence intervals of the ICCs. Finally, an example of case-vignette study is presented. All calculations were carried out using the SAS system