A Study on Machine Learning with Radial Basis Function Networks

本論文主要是討論一系列使用RBF類神經網路 (Radial Basis Function Networks) 在機器學習 (Machine Learning) 領域的研究。 論文的第一個部分討論到如何用正規化程序 (regularization procedure) 有效率的建立一個RBF類神經網路。其中有兩個重要的問題，第一個問題是如何決定核心函數 (kernel function) 的個數與位置。第二個問題是如何決定核心函數組合成RBF類神經網路的權重。在這個論文，我用隨機與遞增式學習兩種方法來決定核心函數的位置，然後用正規化方法與 Cholesky 分解來決定各個核心函數的權重。實驗結果顯示這篇論文所使用的方法可以使用在機器學習與生物資訊領域，而且獲得不錯的結果。 第二個部分討論到一個新的核心密度預測 (kernel density estimation) 的演算法跟他的應用。這個演算法稱為 RVKDE (relaxed variable kernel density estimation) 是由我們實驗室團隊在近期所提出，並應用在許多方面，而我把他拿來應用在機器學習領域的兩個題目上，構成了這個論文的第二部分。This thesis reports a series of studies on machine learning with the radial basis function network (RBFN). The first part of this thesis discusses how to construct an RBFN efficiently with the regularization procedure. In fact, construction of an RBFN with the regularization procedure involves two main issues. The first issue concerns the number of hidden nodes to be incorporated and where the centers of the associated kernel functions should be located. The second issue concerns how the links between the hidden layer and the output layer should be weighted. For the first issue, this thesis discusses the effects with a random samples based approach and an incremental clustering based approach. For the second issue, this thesis elaborates the effects with the Cholesky decomposition employed. Experimental results show that an RBFN constructed with the approaches proposed in this thesis is able to deliver the same level of classification accuracy as the SVM and offers several important advantages. Finally, this thesis reports the experimental results with the QuickRBF package, which has been developed based on the approaches proposed in this thesis, applied to bioinformatics problems. The second study reported in this thesis concerns how the novel relaxed variable kernel density estimation (RVKDE) algorithm that our research team has recently proposed performs in data classification applications. The experimental results reveal that the classifier configured with the RVKDE algorithm is capable of delivering the same level of accuracy as the SVM, while enjoying some advantages in comparison with the SVM. In particular, the time complexity for construction of a classifier with the RVKDE algorithm is O(nlogn), where n is the number of samples in the training data set. This means that it is highly efficient to construct a classifier with the RVKDE algorithm, in comparison with the SVM algorithm. Furthermore, the RVKDE based classifier is able to carry out data classification with more than two classes of samples in one single run. In other words, it does not need to invoke mechanisms such as one-against-one or one-against-all for handling data sets with more than two classes of samples. The successful experiences with the RVKDE algorithm in data classification applications then motivate the study presented next in this thesis. In Section 4.3, a RVKDE based data reduction approach for expediting the model selection process of the SVM is described. Experimental results show that, in comparison with the existing approaches, the data reduction based approach proposed in this thesis is able to expedite the model selection process by a larger degree and cause a smaller degradation of prediction accuracy.ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v CHAPTER I. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Machine Learning Algorithms for RBFN . . . . . . . . . . . . 3 1.3 The Organization of This Thesis . . . . . . . . . . . . . . . . 4 II. An Overview of Radial Basis Function Networks . . . . . . . 7 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Exact Interpolation . . . . . . . . . . . . . . . . . . . . . . . 9 2.3 Function Approximation . . . . . . . . . . . . . . . . . . . . . 10 2.4 Two-stage Training . . . . . . . . . . . . . . . . . . . . . . . 11 III. QuickRBF and Related Works . . . . . . . . . . . . . . . . . . . 12 3.1 An Efficient Regularization Mechanism for Radial Basis Function Networks . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.1.1 Traditional Least Mean Square Error Method . . . . 14 3.1.2 Proposed Least Mean Square Error Method with Statistics Techniques . . . . . . . . . . . . . . . . . 15 3.2 Construction of Radial Basis Function Networks with an Incremental Clustering Algorithm . . . . . . . . . . . . . . . . . 18 3.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . 18 3.2.2 Determining the Centers . . . . . . . . . . . . . . . 21 3.2.3 Calculation of the Bandwidths . . . . . . . . . . . . 24 3.2.4 Calculation of theWeights . . . . . . . . . . . . . . 24 3.2.5 Experiments on Data Classification Data Sets . . . 25 3.2.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . 28 3.3 QuickRBF : A Radial Basis Function Network based Data Classification Package . . . . . . . . . . . . . . . . . . . . . . 29 3.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . 30 3.3.2 Design and Implementation of the Learning Algorithm 30 3.3.3 Experiments on Data Classification Benchmark Data Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.3.4 Experiments on Protein Secondary Structure Prediction Data Sets . . . . . . . . . . . . . . . . . . . 35 IV. The Relaxed Variable Kernel Density Estimation Algorithm and Its Applications . . . . . . . . . . . . . . . . . . . . . . . . . 38 4.1 The Relaxed Variable Kernel Density Estimation Algorithm . 38 4.1.1 Development of the Relaxed Variable Kernel Density Estimation Algorithm . . . . . . . . . . . . . . . . . 38 4.1.2 Properties of the Relaxed Variable Kernel Density Estimation . . . . . . . . . . . . . . . . . . . . . . . 44 4.2 A RVKDE based Data Classifier . . . . . . . . . . . . . . . . 46 4.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . 46 4.2.2 RelatedWorks . . . . . . . . . . . . . . . . . . . . . 49 4.2.3 Overview of Data Classification with the RVKDE based Algorithm . . . . . . . . . . . . . . . . . . . . 51 4.2.4 Implementation Issues and Analysis of Time Complexity . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.2.5 Experimental Results and Discussions . . . . . . . . 57 4.3 Expediting SVM Model Selection Process with the RVKDE Alogorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 4.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . 64 4.3.2 Model Selection for Support Vector Machines . . . . 67 4.3.3 The RVKDE Based Data Reduction Mechanism . . 68 4.3.4 Experimental Results . . . . . . . . . . . . . . . . . 71 V. Discussions and Conclusions . . . . . . . . . . . . . . . . . . . . 74 BIBLIOGRAPHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7