Accurate and Interpretable Regression Trees using Oracle Coaching

In many real-world scenarios, predictive modelsneed to be interpretable, thus ruling out many machine learningtechniques known to produce very accurate models, e.g., neuralnetworks, support vector machines and all ensemble schemes.Most often, tree models or rule sets are used instead, typicallyresulting in significantly lower predictive performance. The over-all purpose of oracle coaching is to reduce this accuracy vs.comprehensibility trade-off by producing interpretable modelsoptimized for the specific production set at hand. The methodrequires production set inputs to be present when generating thepredictive model, a demand fulfilled in most, but not all, predic-tive modeling scenarios. In oracle coaching, a highly accurate, butopaque, model is first induced from the training data. This model(“the oracle”) is then used to label both the training instances andthe production instances. Finally, interpretable models are trainedusing different combinations of the resulting data sets. In thispaper, the oracle coaching produces regression trees, using neuralnetworks and random forests as oracles. The experiments, using32 publicly available data sets, show that the oracle coachingleads to significantly improved predictive performance, comparedto standard induction. In addition, it is also shown that ahighly accurate opaque model can be successfully used as a pre-processing step to reduce the noise typically present in data, evenin situations where production inputs are not available. In fact,just augmenting or replacing training data with another copyof the training set, but with the predictions from the opaquemodel as targets, produced significantly more accurate and/ormore compact regression trees.Sponsorship:This work was supported by the Swedish Foundation for StrategicResearch through the project High-Performance Data Mining for Drug EffectDetection (IIS11-0053), the Swedish Retail and Wholesale DevelopmentCouncil through the project Innovative Business Intelligence Tools (2013:5)and the Knowledge Foundation through the project Big Data Analytics byOnline Ensemble Learning (20120192).