AutoML technologies for the identification of sparse classification and outlier detection models

Liuliakov A, Hermes L, Hammer B. AutoML technologies for the identification of sparse classification and outlier detection models. Applied Soft Computing. 2023;133: 109942.Automated machine learning (AutoML) technologies offer powerful methods to automate the choice of meta-parameters and the instantiations of components of the machine learning training pipelines, such as an optimum form of data preprocessing or a suitable strength of model regularization. Besides given training data, AutoML relies on a suitable learning objective or scoring function and a search space in which to optimize the choices. Currently, most AutoML technologies focus on a single objective, which is related to the expected accuracy of the found model as evaluated according to the chosen learning objective. Additional desired characteristics such as model sparsity for an increased model efficiency and interpretability can be integrated as additional penalty terms in the objective function. Yet, this leads to one solution only, and does not mediate in between accuracy and sparsity as two usually contradicting objectives. In this contribution, we are interested in AutoML technologies which explore the full Pareto-front of sparse versus accurate models rather than a single average only. Since it is not guaranteed that architectural and meta-parameter choices stay constant along the full Pareto-front, averaging the two objectives is not necessarily optimal in this realm. Hence we propose how to treat this challenge by a novel iterative pipeline, which combines an AutoML method with feature selection technologies. We compare this technology to alternatives including baselines in two relevant modeling tasks, classification and outlier detection. We demonstrate the performance of these strategies for a couple of benchmark tasks.