Least Squares Support Vector Regression with Applications to Large-Scale Data: a Statistical Approach (Kleinste kwadraten support vector regressie met toepassingen op grote datasets: een statistische benadering)

Nonparametric regression is a very popular tool for data analysis because thesetechniques impose few assumptions about the shape of the mean function. Hence,they are extremely flexible tools for uncovering nonlinear relationships betweenvariables. A disadvantage of these methods is their computational complexitywhen considering large data sets. In order to reduce the complexity for leastsquares support vector machines (LS-SVM), we propose a method called Fixed-Size LS-SVM which is capable of handling large data set on standard personalcomputers.We study the properties of the LS-SVM regression when relaxing the Gauss-Markov conditions. We propose a robust version of LS-SVM based on iterativereweighting with weights based on the distribution of the error variables. We showthat the empirical maxbias of the proposed robust estimator increases slightly withthe number of outliers in region and stays bounded right up to the breakdownpoint. We also establish three conditions to obtain a fully robust nonparametricestimator.We investigate the consequences when the i.i.d. assumptions is violated. Weshow that, for nonparametric kernel based regression, classical model selectionprocedures such as cross-validation, generalized cross-validation and v-fold crossvalidationbreak down in the presence of correlated data and not the chosensmoothing method. Therefore, we develop a model selection procedure for LSSVMin order to effectively handle correlation in the data without requiring anyprior knowledge about the correlation structure.Next, we propose bias-corrected 100(1 − α)% approximate confidence andprediction intervals (pointwise and uniform) for linear smoothers, in particularlyfor LS-SVM. We prove, under certain conditions, the asymptotic normality of LSSVM.Further, we show the practical use of these interval estimates by means oftoy examples for regression and classification.Finally, we illustrate the capabilities of the proposed methods on a number ofapplications i.e. system identification, hypothesis testing and density estimation.1 Introduction 1 1.1 Historical Evolution and General Background 1.2 Practical Applications 1.2.1 Biomedical Data 1.2.2 Financial Data 1.2.3 System Identification 1.2.4 Time Series Analysis 1.3 Organization and Contributions of the Thesis 2 Model Building 2.1 Regression Analysis and Loss Functions 2.2 Assumptions, Restrictions and Slow Rate 2.3 Curse of Dimensionality 2.4 Parametric and Nonparametric Regression Estimators: An Overview 2.4.1 Parametric Modeling 2.4.2 Local Averaging 2.4.3 Local Modeling 2.4.4 Global Modeling 2.4.5 Penalized Modeling 2.5 Support Vector Machines 2.5.1 Basic Idea of Support Vector Machines 2.5.2 Primal-Dual Formulation of Support Vector Machines 2.5.3 Least Squares Support Vector Machines 2.6 Conclusions 3 Model Selection Methods 3.1 Introduction 3.2 Cross-Validation Procedures 3.2.1 Cross-Validation Philosophy 3.2.2 Leave-One-Out Cross-Validation 3.2.3 v-fold Cross-Validation 3.2.4 Generalized Cross-Validation 3.3 Complexity Criteria: Final Prediction Error, AIC, Mallows’ Cp and BIC 3.4 Choosing the Learning Parameters 3.4.1 General Remarks 3.4.2 Optimization Strategy 3.5 Conclusions 4 Fixed-Size LS-SVM 4.1 Introduction 4.2 Estimation in the Primal Space 4.2.1 Finite Approximation to the Feature Map 4.2.2 Solving the Problem in Primal Space 4.3 Active Selection of a Subsample 4.3.1 Subsample Based on Entropy Criteria 4.3.2 Bandwidth Selection for Density Estimation 4.3.3 Solve-the-Equation Plug-In Method 4.3.4 Maximizing R´enyi Entropy vs. Random Sampling 4.4 Selecting the number of prototype vectors 4.5 Fast v-fold Cross-Validation for FS-LSSVM 4.5.1 Extended Feature Matrix Can Fit Into Memory 4.5.2 Extended Feature Matrix Cannot Fit Into Memory 4.6 Computational Complexity and Numerical Experiments on v-fold CV 4.6.1 Computational Complexity Analysis 4.6.2 Numerical Experiments 4.7 Classification and Regression Results 4.7.1 Description of the Data Sets 4.7.2 Description of the Reference Algorithms 4.7.3 Performance of binary FS-LSSVM classifiers 4.7.4 Performance of multi-class FS-LSSVM classifiers 4.7.5 Performance of FS-LSSVM for Regression 4.8 Conclusions 5 Robustness in Kernel Based Regression 5.1 Introduction 5.2 Measures of Robustness 5.2.1 Influence Functions and Breakdown Points 5.2.2 Empirical Influence Functions 5.3 Residuals and Outliers in Regression 5.3.1 Linear Regression 5.3.2 Kernel Based Regression 5.4 Robustifying LS Kernel Based Regression 5.4.1 Problems with Outliers in Nonparametric Regression 5.4.2 Theoretical Background 5.4.3 Application to Least Squares Support Vector Machines 5.4.4 Weight Functions 5.4.5 Speed of Convergence-Robustness Tradeoff 5.4.6 Robust Selection of Tuning Parameters 5.5 Simulations 5.5.1 Empirical Maxbias Curve 5.5.2 Toy example 5.5.3 Real Life Data Sets 5.6 Conclusions 6 Kernel Regression with Correlated Errors 6.1 Introduction 6.2 Problems with Correlation 6.3 New Developments in Kernel Regression with Correlated Errors 6.3.1 No Positive Definite Kernel Constraint 6.3.2 Positive Definite Kernel Constraint 6.3.3 Drawback of Using Bimodal Kernels 6.4 Simulations 6.4.1 CC-CV vs. LOO-CV with Different Noise Models 6.4.2 Evolution of the Bandwidth Under Correlation 6.4.3 Comparison of Different Bimodal Kernels 6.4.4 Real life data sets 6.5 Conclusions 7 Confidence and Prediction Intervals 7.1 Introduction 7.2 Estimation of Bias and Variance 7.2.1 LS-SVM Regression and Smoother Matrix 7.2.2 Bias Estimation 7.2.3 Variance Estimation 7.3 Confidence and Prediction Intervals: Regression 7.3.1 Pointwise Confidence Intervals 7.3.2 Simultaneous Confidence Intervals 7.3.3 Pointwise and Simultaneous Prediction Intervals 7.4 Bootstrap Based Confidence and Prediction Intervals 7.4.1 Bootstrap Based on Residuals 7.4.2 Construction of Bootstrap Confidence and Prediction Intervals 7.5 Simulations: The Regression Case 7.5.1 Empirical Coverage Probability 7.5.2 Homoscedastic Examples 7.5.3 Heteroscedastic Examples and Error Variance Estimation 7.6 Confidence Intervals: Classification 7.6.1 Classification vs. Regression 7.6.2 Illustration and Interpretation of the Method 7.6.3 Simulations: The Classification Case 7.7 Conclusions 8 Applications and Case Studies 8.1 System Identification with LS-SVMLab 8.1.1 General Information 8.1.2 Model Identification 8.2 SYSID 2009: Wiener-Hammerstein Benchmark 8.2.1 Model Structure 8.2.2 Data Description and Training Procedure 8.2.3 Estimation and Model Selection 8.2.4 Results on Test Data 8.3 Nonparametric Comparison of Densities Based on Statistical Bootstrap 8.3.1 Introduction to the Problem 8.3.2 Kernel Density Estimation 8.3.3 Formulation and Construction of the Hypothesis Test 8.3.4 Illustrative Examples 8.4 Finding the Maximum in Hysteresis Curves 8.5 Conclusions 9 Summary, Conclusions and Future Research 9.1 Summary and Main Conclusions 9.2 Future Research A Coupled Simulated Annealing References Curriculum vitaenrpages: 246status: publishe