Design of a monitoring system for distributed data processing pipelines : ECiDA platform monitoring

Distributed data processing pipelines allow businesses to reduce the time needed to get models in production, fostering accelerated innovation and breakthroughs in many fields. Monitoring is crucial for ensuring the health and optimal performance of these systems. This thesis aims to provide a blueprint for designing a monitoring system for distributed data processing systems, specifically targeting the ECiDA platform, which facilitates the design and deployment of such pipelines. A literature and documentation review was done to define how these systems should be monitored. Then, a software architecture for the monitoring system was defined and, by reviewing and comparing available open-source tools, a possible implementation of the defined architecture was proposed. It has been observed that monitoring is highly dependent on the specific use case, problem nature, data format, and model type, among other factors. However, the proposed architecture can serve as a foundation for developing a monitoring system for systems like ECiDA, with minor implementation details varying based on each specific problem and contex