A scalable platform for monitoring data intensive applications

Latest advances in information technology and the widespread growth in different areas are producing large amounts of data. Consequently, in the past decade a large number of distributed platforms for storing and processing large datasets have been proposed. Whether in development or in production, monitoring the applications running on these platforms is not an easy task, dedicated tools and platforms were proposed for this task yet none are specially designed for Big Data frameworks. In this paper we present a distributed, scalable, highly available platform able to collect, store, query and process monitoring data obtained from multiple Big Data frameworks. Alongside the architecture we experimentally show that the solution proposed is scalable and can handle a substantial quantity of monitoring data.This work has received funding from the EC-funded project H2020 DICE (Agreement 644869), which aims at providing a toolchain that makes the task of developing Big Data applications less daunting and the H2020 ASPIDE project (Agreement 801091). This work was partially supported by grants from Romanian Ministry of Research and Innovation, grant Acronim (PNIII-P4-ID-PCE-2016-0842) and grant BID (PNIII-P1-PDI-PFE-2018-028)