Automating Model Search for Large Scale Machine Learning

The proliferation of massive datasets combined with the develop-ment of sophisticated analytical techniques has enabled a wide va-riety of novel applications such as improved product recommenda-tions, automatic image tagging, and improved speech-driven inter-faces. A major obstacle to supporting these predictive applications is the challenging and expensive process of identifying and train-ing an appropriate predictive model. Recent efforts aiming to au-tomate this process have focused on single node implementations and have assumed that model training itself is a black box, limit-ing their usefulness for applications driven by large-scale datasets. In this work, we build upon these recent efforts and propose an architecture for automatic machine learning at scale comprised of a cost-based cluster resource allocation estimator, advanced hyper-parameter tuning techniques, bandit resource allocation via runtime algorithm introspection, and physical optimization via batching and optimal resource allocation. The result is TUPAQ, a component of the MLbase system that automatically finds and trains models for a user’s predictive application with comparable quality to those found using exhaustive strategies, but an order of magnitude more efficiently than the standard baseline approach. TUPAQ scales to models trained on Terabytes of data across hundreds of machines. 1