New Nonlinear Machine Learning Algorithms with Theoretical Analysis

Recent advances in machine learning have spawned progress in various fields. In the context of machine learning data, nonlinearity is an intrinsic property. Therefore, a nonlin- ear model will facilitate the flexibility of representation and fit the data properly. However, increasing flexibility usually means the higher complexity and less interpretability. Thus, there is a niche for designing feasible nonlinear machine learning models to handle the fore- mentioned challenges. As a part of this work, a new method, called as sparse shrunk additive models (SSAM) is proposed. This model explores the structure information among features for high-dimensional nonparametric regression with the allowance of the flexible interactions among features. It bridges the sparse kernel regression and sparse feature selection. Theoretical results on the convergence rate and sparsity characteristics are established by the novel analysis techniques with integral operator and concentration estimate. Most of the nonlinear models usually involve tuning multiple (up to thousands) hyper- parameters, which plays a pivotal role in model generalization. Another part of this work is a new hyperparameter optimization method with zeroth-order hyper-gradients (HOZOG). We proved the feasibility analysis of using HOZOG to achieve hyperparameter optimization under the condition of Lipschitz continuity. The extensive experiments verify the analysis. For large-scale data, there remain computational challenges in implementing various al- gorithms. To address this issue, we propose a new regularized modal regression model with robust sampling strategy. Unlike conventional sampling for large-scale least squares, our sampling probabilities are dependent on the robust loss function for learning the conditional mode. We provide theoretical analysis to support the proposed model: the approximation bound is established by error analysis with Rademacher complexity, and the robustness characterization is provided based on the finite sample breakdown point analysis. The ex- periments are conducted on both synthetic and real-world data sets and the empirical results demonstrate the promising performance of resulting estimator