On the Testing and Estimation of High-Dimensional Covariance Matrices

Many applications of modern science involve a large number of parameters. In many cases, the number of parameters, p, exceeds the number of observations, N. Classical multivariate statistics are based on the assumption that the number of parameters is fixed and the number of observations is large. Many of the classical techniques perform poorly, or are degenerate, in high-dimensional situations. In this work, we discuss and develop statistical methods for inference of data in which the number of parameters exceeds the number of observations. Specifically we look at the problems of hypothesis testing regarding and the estimation of the covariance matrix. A new test statistic is developed for testing the hypothesis that the covariance matrix is proportional to the identity. Simulations show this newly defined test is asymptotically comparable to those in the literature. Furthermore, it appears to perform better than those in the literature under certain alternative hypotheses. A new set of Stein-type shrinkage estimators are introduced for estimating the covariance matrix in large-dimensions. Simulations show that under the assumption of normality of the data, the new estimators are comparable to those in the literature. Simulations also indicate the new estimators perform better than those in the literature in cases of extreme high-dimensions. A data analysis of DNA microarray data also appears to confirm our results of improved performance in the case of extreme high-dimensionality