Input Warping for Bayesian Optimization of Non-stationary Functions

Bayesian optimization has proven to be a highly effective methodology for the global optimization of unknown, expensive and multimodal functions. The ability to accurately model distributions over functions is critical to the effectiveness of Bayesian optimization. Although Gaussian processes provide a flexible prior over functions which can be queried efficiently, there are various classes of functions that remain difficult to model. One of the most frequently occurring of these is the class of non-stationary functions. The optimization of the hyperparameters of machine learning algorithms is a problem domain in which parameters are of-ten manually transformed a-priori, for example by optimizing in “log-space”, to mitigate the effects of extreme non-stationarity. We develop a methodology for automatically learning monotonic, bijective transformations or warpings of the input space using the beta cumulative distribution function. Marginalizing over the parameters of the of the beta distribution allows the Bayesian optimization to integrate over a wide class of flexible warping functions. We demonstrate on four challenging machine learning problems that the optimization converges to significantly better solutions much faster when this input warping is used.