Efficient Learning Algorithms for Training Large-Scale and High-Dimensional Machine Learning Models

Machine learning has achieved tremendous successes and played increasingly essential roles in many application scenarios in the past decades. The recent advance in machine learning relies heavily upon the emergence of big data with both massive samples and numerous features. However, the computational inefficiency and memory burden of the learning algorithms restrict the capability of machine learning for large-scale applications. Therefore, it is important to design efficient learning algorithms for big data mining. In this dissertation, we propose several newly designed efficient learning algorithms to address the challenges of high dimensionality from the aspects of both samples and features for big data mining. First, we develop an efficient approximate solution path algorithm and introduce a safe screening rule to accelerate the model training of the Ordered Weighted L1 regression. We also formulate a unified safe variable screening rule for the family of ordered weighted sparse models, which can effectively accelerate the training algorithms. Second, we develop a new accelerated doubly stochastic gradient descent method for regularized loss minimization problems. The proposed method is able to simultaneously achieve a linear convergence rate and linear rate of explicit model identification. Finally, we design a novel distributed dynamic safe screening method in parallel and distributed computing to solve sparse models and apply the method to the shared-memory and distributed-memory architecture. The method can accelerate the training process without any loss of accuracy. The contributions of the thesis are expected to speed up the training of large-scale machine learning models through smart handling of the model sparsity and data sparsity