Constructing and Predicting School Advice for Academic Achievement: A Comparison of Item Response Theory and Machine Learning Techniques

Educational tests can be used to estimate pupils’ abilities and thereby give an indication of whether their school type is suitable for them. However, tests in education are usually conducted for each content area separately which makes it difficult to combine these results into one single school advice. To this end, we provide a comparison between both domain-specific and domain-agnostic methods for predicting school advice. Both use data from a pupil monitoring system in the Netherlands, which keeps track of pupils’ educational progress over several years by a series of tests measuring multiple skills. An IRT model is calibrated from which an ability score is extracted and is subsequently plugged into a multinomial log- linear regression model. Second, we train a random forest (RF) and a shallow neural network (NN) and apply case weighting to give extra attention to pupils who switched between school types. When considering the performance of all pupils, RFs provided the most accurate predictions followed by NNs and IRT respectively. When only looking at the performance of pupils who switched school type, IRT performed best followed by NNs and RFs. Case weighting proved to provide a major improvement for this group. Lastly, IRT was found to be much easier to explain in comparison to the other models. Thus, while ML provided more accurate results, this comes at the cost of a lower explainability in comparison to IRT