Rule Extraction from Support Vector Machines: A Geometric Approach

Despite the success of connectionist systems in prediction and classi¯cation problems, critics argue that the lack of symbol processing and explanation capability makes them less competitive than symbolic systems. Rule extraction from neural networks makes the interpretation of the behaviour of connectionist networks possible by relating sub-symbolic and symbolic process- ing. However, most rule extraction methods focus only on speci¯c neural network architectures and present limited generalization performance. Support Vector Machine is an unsupervised learning method that has been recently applied successfully in many areas, and o®ers excellent generalization ability in comparison with other neural network, statistical, or symbolic machine learning models. In this thesis, an algorithm called Geometric and Oracle-Based Support Vector Machines Rule Extraction (GOSE) has been proposed to overcome the limitations of other rule-extraction methods by extracting comprehensible models from Support Vector Machines (SVM). This algorithm views the extraction as a geometric task. Given a trained SVM network, GOSE queries the synthetic instances and draws conjunction rules by approximating the optimization problem. The extracted rule set also represents the approximation of the SVM classi¯cation boundary. Unlike previous works in SVM rule-extraction, GOSE is broadly applicable to different networks and problems because it need not rely on training examples and network architectures. Theoretical proof guarantees that GOSE is capable of approximating the behavior of SVM networks. Empirical experiments are conducted on di®erent SVM networks from binary classification networks to multi-class networks in various classi¯cation domains. The result of experiments demonstrates that GOSE can extract comprehensible rules with high levels of accuracy and ¯delity for their corresponding networks. GOSE also exhibits superior consistency. After analyzing and applying several optimizing measures, the complexity of GOSE was improved. In brief, GOSE provides a novel way to explain how an SVM network functions