Scalable Collaborative Caching and Storage Platform for Data Analytics

The emerging Big Data ecosystem has brought about dramatic proliferation of paradigms for analytics. In the race for the best performance, each new engine enforces tight coupling of analytics execution with caching and storage functionalities. This one-for-all approach has led to either oversimplifications where traditional functionality was dropped or more configuration options that created more confusion about optimal settings. We avoid user confusion by following an integrated multi-service approach where we assign responsibilities to decoupled services. In our solution, called Gluon, we build a collaborative cache tier that connects state-of-art analytics engines with a variety of storage systems. We use both open-source and proprietary technologies to implement our architecture. We show that Gluon caching can achieve 2.5x-3x speedup when compared to uncustomized Spark caching while displaying higher resource utilization efficiency. Finally, we show how Gluon can integrate traditional storage back-ends without significant performance loss when compared to vanilla analytics setups.M.A.S