Should Colleges Invest in Machine Learning? Comparing the Predictive Powers of Early Momentum Metrics and Machine Learning for Community College Credential Completion

Among community college leaders and others interested in reforms to improve student success, there is growing interest in adopting machine learning techniques to predict credential completion. However, machine learning algorithms are often complex and are not readily accessible to practitioners, for whom a simpler set of near-term measures may serve as sufficient predictors. This study compares the out-of-sample predictive power of early momentum metrics —13 near-term success measures suggested by the literature—with that of metrics from machine-learning-based models that employ approximately 500 predictors for community college credential completion. Using transcript data from approximately 50,000 students at more than 30 community colleges in two states, the author finds that the early momentum metrics that were modeled by logistic regression accurately predict completion for approximately 80% of students. This classification performance is comparable to that of the machine-learning-based models. The early momentum metrics even outperform the machine-learning-based models in probability estimation. These findings suggest that early momentum metrics are useful predictors for credential completion and that the marginal gain from using a machine-learning-based model over early momentum metrics is typically small