Elementary Estimators for High-Dimensional Linear Regression

We consider the problem of structurally con-strained high-dimensional linear regression. This has attracted considerable attention over the last decade, with state of the art statistical estimators based on solving regularized convex programs. While these typically non-smooth convex pro-grams can be solved by the state of the art op-timization methods in polynomial time, scaling them to very large-scale problems is an ongoing and rich area of research. In this paper, we at-tempt to address this scaling issue at the source, by asking whether one can build simpler possibly closed-form estimators, that yet come with statis-tical guarantees that are nonetheless comparable to regularized likelihood estimators. We answer this question in the affirmative, with variants of the classical ridge and OLS (ordinary least squares estimators) for linear regression. We an-alyze our estimators in the high-dimensional set-ting, and moreover provide empirical corrobora-tion of its performance on simulated as well as real world microarray data. 1