Mixed models in quantitative trait loci and association mapping with bootstrap thresholds

Understanding the complicated process by which the genetic code is translated into the phenotypic expression of complex traits has long been a goal of genetics research. Associating regions on the genome with complex traits has led to improved treatments for disease and programs for breeding animals and crops. Quantitative trait loci (QTL) and association mapping are effective analytic methodologies for locating regions of the genome that are associated with complex traits. Due to the intrinsic relationships and unknown interactions between phenotypic data and the genome, thresholds for identifying significant genetic locations that account for experimental conditions have been difficult to derive. Permutation testing has successfully provided significance thresholds in QTL analysis studies; however, due to the requirement of exchangeability, permutation thresholds are applicable only to simple linear models, which limits the ability to locate associations between phenotypic data and the genome because biologically important cofactors can not be incorporated into the analysis. Two models, a mixture of mixed models and a mixed mixture model, are developed that extend QTL interval mapping methods to include mixed models that can incorporate biological cofactors. Novel applications of the alternating expectation-conditional maximization and nested EM algorithm are developed that provide maximum likelihood estimates for a mixture of mixed models and mixed mixture model, respectively. A bootstrap threshold algorithm is derived that establishes significance thresholds that are appropriate for a mixed model QTL analysis and is extended to association mapping studies that incorporate mixed models to control for population structure. In simulation and real data studies, the proposed mixed models for QTL mapping demonstrate improved detection of additive QTL effects when influential covariates are incorporated into the analysis. Additionally, it is shown empirically that the bootstrap threshold algorithm establishes appropriate thresholds for mixed model QTL or association mapping analyses