Using Genetic Programming to Obtain Implicit Diversity

When performing predictive data mining, the useof ensembles is known to increase prediction accuracy,compared to single models. To obtain this higher accuracy,ensembles should be built from base classifiers that are bothaccurate and diverse. The question of how to balance these twoproperties in order to maximize ensemble accuracy is, however,far from solved and many different techniques for obtainingensemble diversity exist. One such technique is bagging, whereimplicit diversity is introduced by training base classifiers ondifferent subsets of available data instances, thus resulting inless accurate, but diverse base classifiers. In this paper, geneticprogramming is used as an alternative method to obtainimplicit diversity in ensembles by evolving accurate, butdifferent base classifiers in the form of decision trees, thusexploiting the inherent inconsistency of genetic programming.The experiments show that the GP approach outperformsstandard bagging of decision trees, obtaining significantlyhigher ensemble accuracy over 25 UCI datasets. This superiorperformance stems from base classifiers having both higheraverage accuracy and more diversity. Implicitly introducingdiversity using GP thus works very well, since evolved baseclassifiers tend to be highly accurate and diverse