Understanding Why: Statistical Techniques to Infer Causality are Underused in Computing Education Research

A common thread through education research is asking questions about how treatments applied to students affect their education, career, and other outcomes. For example: Will being taught in a certain way increase students' learning? Will taking a computer science course lead to higher job satisfaction in the future? Are remedial programs serving their intended purpose? The most robust way to establish the causal effects of treatments is to perform randomized controlled trials. However, in the context of education, it would frequently be unethical or logistically impossible to simply assign students to take a certain class or participate in a certain program for the purpose of research. As a result, we often take advantage of natural experiments or quasi-experiments. In such situations, the traditional method of analysis is to look at the correlation between the treatment and outcome variables. However, this doesn't tell us whether the outcome was caused by the treatment, as there are almost always substantial selection biases or confounding variables. In the past few decades, advanced statistical methods have been developed to analyze the assignment of subjects to treatments as if it was random, allowing us to deduce the causal effect of the treatment. Such methods include difference-in-differences, instrumental variables, and regression discontinuity design. In this paper we argue that these methods have been underused in computing education research. To encourage their increased use, we describe the methods and present selected examples of education studies where they have allowed researchers to bridge the gap from correlation to causation