Experimental methods for model selection

Models need to be complex to cope with the complexity of today’s data. Model complexity arises in part from model, or tuning, parameters that either determine a model from a class of models, or that specify options in a learning algorithm that searches for structure in the data. For example, the following are tuning parameters in locally-weighted regression: bandwidth (α), polynomial degree (λ), scaling coefficients of the regressors (γ1…, γk), and depth of k-d tree used for fast computation. The selection of such tuning parameters is a ubiquitous task in statistics and machine learning. The standard approach is an automatic machine method: optimize a model selection criterion such as the cross-validation sum of squares by searching the space of tuning parameters with an algorithm for unconstrained optimization and choosing the values of those parameters which minimize the criterion. In place of machine minimization, this work describes an experimental approach that regards the tuning parameters as explanatory variables in a computer experiment (Santner et al., 2003). The methodology of experimental design plays a role; the analyst designs combinations of tuning parameter values over which to run this multi-factor multi-response experiment. The responses are the selection criterion, a complexity measure and the estimated noise variance. These meta-data are subject to techniques used for analyzing designed experiments: extensive data visualization, transforming the factors to simplify the response surfaces, and evolving the design space toward a region of better performance. The experimental approach can succeed with more tuning parameters than would a machine optimization. Incorporating human guidance also allows model selection to be tailored to the particular needs of the analysis, for example, a balancing of bias and variance. Often, with experience across data sets, much insight is gained into the effect and relationship of the tuning parameters, which can be used to improve the model or learning algorithm. These benefits are shown with an application of the approach to locally-weighted regression. Routines implementing the methodology are available from the author, including updates to the loess code