High-Dimensional Methods and Inference

D ata with a large number of variables relative to the sample size—“high-dimensional data”—are readily available and increasingly common in empirical economics. High-dimensional data arise through a combination of two phenomena. First, the data may be inherently high dimensional in that many different char-acteristics per observation are available. For example, the US Census, the Current Population Survey, the Survey of Income and Program Participation, the National Longitudinal Survey of Youth, and the American Housing Survey collect informa-tion on hundreds of individual characteristics. Economists are also increasingly using scanner datasets that record transaction-level data for households across a wide range of products, or text data where counts of words in documents may be used as variables. In both of these latter examples, there may be thousands or tens of thousands of available variables per observation. Second, even when the number of available variables is relatively small, researchers rarely know the exact functional form with which the small number of variables enters the model of interest. Researchers are thus faced with a large set of potential variables formed by different ways of interacting and transforming the underlying variables