Covariate balance assessment, model selection and bias in propensity score matching: A simulation study

Background: In building propensity score (PS) model, inclusion of interaction/square terms in addition to the main terms and the use of balance measures has been suggested. However, the impact of assessing balance of several sets of covariates and their interactions/squares on bias/precision is not well studied. Objectives: The aim of this study was to investigate the impact of balance assessment with respect to different covariates on bias of the estimated treatment effect and PS model selection. Methods: Simulation study was conducted using binary treatment and outcome data, and several covariates: confounding terms, risk factors (RFs; only related to outcome), instrumental variables (IVs; only related to treatment), and their interactions/squares. Treatment effects (risk ratios) were estimated using PS matching, and covariate balance was assessed using standardized difference. PS model selection was based on the balance achieved on different sets of covariates, and their interactions/squares. The types of covariates included in balance assessment were compared with respect to bias/precision of the effect estimate as well as the PS model selected. Results: PS model selection based on balance of confounding variables and RFs provided the least biased estimates. Inclusion of interactions/squares in balance calculation improved the precision of the estimate without increasing the bias. Although PS model selection based on balance calculation on all covariates and on confounding terms as well as IVs resulted in similar estimates in the absence of unmeasured confounding, inclusion of interactions/squares in balance calculation increased the bias (up to 13.6%) while reducing the precision. When PS model was selected based on the balance achieved only on confounding terms, the PS model containing only confounding terms was often selected followed by the PS model with confounding terms and RFs. Conclusions: In PS model selection based on covariate balance, the choice of covariates and interaction/squares for balance calculation has substantial impact on bias/precision of the treatment effect. Researchers should consider PS model selection based on the balance achieved on confounding variables, RFs and important interactions among confounders and RFs