Inference for instrument reliability and rater agreement within a multi-rater and longitudinal data setting

Thesis (Ph. D.)--University of Rochester. School of Medicine & Dentistry. Dept. of Biostatistics and Computational Biology, 2008.Analysis of instrument reliability and rater agreement is used in a wide range of behavioral, biomedical, psychosocial, and health-care related research to assess psychometric properties of instruments, consensus in disease diagnoses, fidelity of psychosocial intervention, and accuracy of proxy outcomes. Cohen’s kappa and the concordance correlation coefficient (CCC) are the most widely used measures of agreement and reliability for categorical and continuous outcomes. In many modern-day applications, data are often clustered and nested, making inference difficult to perform using existing methods. In addition, as longitudinal study designs become increasingly popular, missing data has become a serious issue, and the lack of methods to systematically address this problem has stifled the progress of research in these fields. In this thesis, we develop a uni ed approach based on a new class of U-statistics estimates to concurrently tackle the complexities involved in addressing missing data and other related issues arising from a general multi-rater and longitudinal data setting for inference for both agreement and reliability indices. The approach is illustrated with real data from studies in biomedical as well as psychosocial research