Features of Big Data and sparsest solution in high confidence set

This chapter summarizes some of the unique features of Big Data analysis. These features are shared neither by low-dimensional data nor by small sam-ples. Big Data pose new computational challenges and hold great promises for understanding population heterogeneity as in personalized medicine or ser-vices. High dimensionality introduces spurious correlations, incidental endo-geneity, noise accumulation, and measurement error. These unique features are very distinguished and statistical procedures should be designed with these issues in mind. To illustrate, a method called a sparsest solution in high-confidence set is introduced which is generally applicable to high-dimensional statistical inference. This method, whose properties are briefly examined, is natural as the information about parameters contained in the data is summa-rized by high-confident sets and the sparsest solution is a way to deal with the noise accumulation issue. The first decade of this century has seen the explosion of data collection i