Unobservable Selection and Coefficient Stability: Theory and Validation

Inferring causal treatment effects in the presence of possible omitted variable bias is as well-known problem. Altonji, Elder and Taber (2005) suggest that the degree of selection on observable variables might be used as a guide to the remaining bias in controlled regressions. I expand on their setup and demonstrate how, with an equal selection assumption, a causal effect can be recovered using coefficients, R-squared values from controlled and uncontrolled regressions and an estimate of the iid noise in the outcome. I discuss the relationship between this technique and the heuristic procedure of adding sequential controls until coefficients stabilize. I consider two validation exercises which explore whether coefficients adjusted in this way are closer to the truth. First, I estimate the impact of early life and prenatal influences on child IQ. Simple controlled regressions give misleading estimates; the bias-adjusted estimates are significantly better. However, I show that the progressive adding of controls in this setting is not effective. Second, I match observational and randomized trial data for 31 treatment-outcome pairs in three public health settings. I estimate a best-fit degree of adjustment across all settings (the free parameter is the amount of iid noise in the outcome). I show that the bias-adjusted coefficients preform much better than simple controlled coefficients; the total error reduction is 30 % and a number of false-positive results are rejected without significant loss in true-positive results. The magnitude of the best-fit adjustment suggests adjusting the controlled coefficient by approximately the same amount as the movement between uncontrolled and controlled coefficients