ORIGINAL ARTICLE Heterogeneity and the Interpretation of Treatment Effect Estimates From Risk Adjustment and Instrumental Variable Methods

Objectives: To contrast the interpretations of treatment effect esti-mates using risk adjustment and instrumental variable (IV) estima-tion methods using observational data when the effects of treatment are heterogeneous across patients. We demonstrate these contrasts by examining the effect of breast conserving surgery plus irradiation (BCSI) relative to mastectomy on early stage breast cancer (ESBC) survival. Methods: We estimated discrete time survival models for 6185 ESBC patients in the 1989–1994 Iowa Cancer Registry via IV estimation using 2 distinct instruments (distance of the patient’s residence from the nearest radiation center, and local area BCSI rate) and controlling for cancer stage, grade, and location; age; comor-bidity; hospital access; payer; diagnosis year; and area poverty level. We then estimated comparable risk adjustment survival models using linear probability methods with robust standard errors. Results: Risk adjustment models yielded average survival estimates similar to trial results. With favorable BCSI selection, these esti-mates represent an upper bound of the true effect for patients receiving BCSI. IV estimates showed a BCSI survival risk for patients whose surgery choices were affected by the instruments and these estimates varied with the instrument specification. Conclusions: When treatment benefits are heterogeneous across patients, treatment effect estimates from observational data can still be useful to policymakers, but they must be interpreted correctly. Risk adjustment methods yield estimates that can assess whether the patients who received treatment benefited from the treatment, but the direction of bias must be considered. In contrast, IV estimates can assess the effect of treatment rate changes, but characteristics of patients whose choices were affected by the instruments must be considered when making such inferences. Key Words: heterogeneity, effectiveness, instrumental variables, risk adjustment, treatment rate