A Bloat-Aware Design for Big Data Applications

Over the past decade, the increasing demands on data-driven busi-ness intelligence have led to the proliferation of large-scale, data-intensive applications that often have huge amounts of data (often at terabyte or petabyte scale) to process. An object-oriented pro-gramming language such as Java is often the developer’s choice for implementing such applications, primarily due to its quick develop-ment cycle and rich community resource. While the use of such lan-guages makes programming easier, significant performance prob-lems can often be seen — the combination of the inefficiencies in-herent in a managed run-time system and the impact of the huge amount of data to be processed in the limited memory space often leads to memory bloat and performance degradation at a surpris-ingly early stage. This paper proposes a bloat-aware design paradigm towards the development of efficient and scalable Big Data applications in object-oriented GC enabled languages. To motivate this work, we first perform a study on the impact of several typical memory bloat patterns. These patterns are summarized from the user complaints on the mailing lists of two widely-used open-source Big Data appli-cations. Next, we discuss our design paradigm to eliminate bloat. Using examples and real-world experience, we demonstrate that programming under this paradigm does not incur significant pro-gramming burden. We have implemented a few common data pro-cessing tasks both using this design and using the conventional object-oriented design. Our experimental results show that this new design paradigm is extremely effective in improving performance — even for the moderate-size data sets processed, we have ob-served 2.5×+ performance gains, and the improvement grows sub-stantially with the size of the data set