High-performance distributed ml at scale through parameter server consistency models

As Machine Learning (ML) applications embrace greater data size and model complexity, practition-ers turn to distributed clusters to satisfy the increased computational and memory demands. Effective use of clusters for ML programs requires considerable exper-tise in writing distributed code, but existing highly-abstracted frameworks like Hadoop that pose low bar-riers to distributed-programming have not, in practice, matched the performance seen in highly specialized and advanced ML implementations. The recent Parameter Server (PS) paradigm is a middle ground between these extremes, allowing easy conversion of single-machine parallel ML programs into distributed ones, while main-taining high throughput through relaxed “consistency models ” that allow asynchronous (and, hence, inconsis-tent) parameter reads. However, due to insufficient the-oretical study, it is not clear which of these consistency models can really ensure correct ML algorithm out-put; at the same time, there remain many theoretically-motivated but undiscovered opportunities to maximize computational throughput. Inspired by this challenge, we study both the theoretical guarantees and empirical behavior of iterative-convergent ML algorithms in ex-isting PS consistency models. We then use the gleaned insights to improve a consistency model using an “ea-ger ” PS communication mechanism, and implement it as a new PS system that enables ML programs to reach their solution more quickly