Cherry Picking with Synthetic Controls

We evaluate whether a lack of guidance on how to choose the matching variables used in the Synthetic Control (SC) estimator creates specification-searching opportunities. We first provide theoretical results showing that specification-searching opportunities would be asymptotically irrelevant when the number of pre-treatment periods goes to infinity when we restrict to a subset of SC specifications. However, based on Monte Carlo simulations and simulations with real datasets, we show significant room for specification searching when the number of pre-treatment periods is finite and when other SC specifications commonly used in SC applications are also considered. This undermines one of the potential advantages of the method, which is providing a transparent way of choosing comparison units and, therefore, being less susceptible to specification searching than alternative methods. To address this problem, we provide recommendations to limit the possibilities for specification searching in the SC method. Finally, we analyze the possibilities for specification searching and our recommendations in two empirical applications