Expressive and explainable rule-based classification

Classification rule learning produces expressive rules so that a human user can easily interpret the rationale behind the predictions of the generated model. Constructing a very accurate classification model may lead to overfitting, a common problem in data mining that causes a leaner to perform badly on test instances. Ensemble learning is a common way to address the problem of overfitting while improving the learner’s accuracy. The idea of ensemble classification is to construct various predictive base learners, and then, combine their predictions. This often goes at the expense of explainability of the predictive model learned, as the analyst is presented with many different classification models. Therefore, predictive learning models are required to be not only reliable and accurate, but also comprehensible to avoid the risk of irreversible misclassification, especially in critical applications such as medical diagnoses, financial analysis, terrorism detection, etc. The level of expressiveness of the individual base learners is one of the most important factors for improving the whole ensemble’s explainablility. Taking this into account, this research focuses on developing a predictive ensemble learner that maintains the expressive power of rule learning models while benefiting from the high predictive performance of ensemble learning. Measuring the expressiveness of a rule-based learner often depends on the complexity of its rule set. A rule set is considered more expressive when it produces fewer number of rules with less complex terms per rule. Also, rule learning approaches can abstain from classification when the algorithm is uncertain about a prediction, which contributes to increased explainablility in the model by ensuring the trustworthiness of the induced rule set. Abstaining is needed to prevent costly false classification. Nevertheless, classifying instances correctly is more desired than abstaining from it in most applications. Therefore, this thesis aims to answer the following raised research question: ‘is it possible to develop a predictive ensemble model, which exhibits a similar expressiveness as the predictive base learner while improving its accuracy and lowering its abstaining rate?’. To achieve that, this thesis makes a number of contributions towards rule induction algorithms in both single-based and ensemble-based systems. Three novel single predictive rule-based algorithms are developed, termed G-Prism-FB, G-Prism-DB, and G-Rules-IQR, where ‘G’ stands for Gaussian distribution. These algorithms are highly expressive on their own, that can be used to serve as the base learners of the ensemble. The results of empirical evaluation show that these algorithms produce expressive and more computationally efficient numeric rule terms compared with frequent discrete intervals. G-Rules-IQR learner, in particular, has shown to be superior in terms of expressiveness and accuracy compared with other rule-based learners. Therefore, it is utilised in this research as a base learning algorithm to induced multiple base classifiers for the ensemble systems. Furthermore, a novel framework for explainable rule-based ensemble algorithms called ReG-Rules (Ranked ensemble G-Rules) is presented. ReG-Rules incorporates three novel methods. First, a new ranking-based approach to rank the base classifiers, and then a selection method to find the best performing models. Second, a new rules merging algorithm to reduce the number of rules induced by each selected model without loss of rule coverage. Third, to reduce the overall number of rules presented to human analyst during the prediction stage, a decision committee of rules is built per classification attempt using a novel weighted voting combination method. Additionally, an extension of ReG-Rules learner termed CRC (Consolidated Rules Construction) is developed. CRC enhances the explainablility of ReG-Rules by generating rules that can be consolidated into a single global rule set and used directly in predictions without the need for a classification committee. The experimental studies show that comparred with the standalone classifier (G-Rues-IQR), ReG-Rules and CRC, are more accurate on all cases. Also, the ReG-Rules and CRC abstaining rates were almost zero on all cases, while the abstaining rate of G-Rules-IQR learner was above 10% in several cases