Using Meta-Learning to Initialize Bayesian Optimization of Hyperparameters

Abstract. Model selection and hyperparameter optimization is cru-cial in applying machine learning to a novel dataset. Recently, a sub-community of machine learning has focused on solving this prob-lem with Sequential Model-based Bayesian Optimization (SMBO), demonstrating substantial successes in many applications. However, for expensive algorithms the computational overhead of hyperpa-rameter optimization can still be prohibitive. In this paper we ex-plore the possibility of speeding up SMBO by transferring knowl-edge from previous optimization runs on similar datasets; specifi-cally, we propose to initialize SMBO with a small number of config-urations suggested by a metalearning procedure. The resulting simple MI-SMBO technique can be trivially applied to any SMBO method, allowing us to perform experiments on two quite different SMBO methods with complementary strengths applied to optimize two ma-chine learning frameworks on 57 classification datasets. We find that our initialization procedure mildly improves the state of the art in low-dimensional hyperparameter optimization and substantially im-proves the state of the art in the more complex problem of combined model selection and hyperparameter optimization.