Correlation and Consolidation of Distributed Logging Data in Enterprise Clouds

Abstract—Due to the availability of virtualization technolo-gies and related cloud infrastructures, the amount and also the complexity of logging data of systems and services grow steadily. Automated correlation and aggregation techniques are required to support a contemporary processing and interpre-tation of relevant logging data. In the past, this was achieved using highly centralized logging systems. Based on this fact, the paper introduces a prototype for an automated semantical correlation, aggregation and condensation of logging information. The prototype relies on a NoSQL storage back-end that is used to persist consolidated messages of distributed logging sources in a highly performant manner. This step of consolidation includes strategies for minimizing long-term storage, and by using correlation techniques also offers possibilities to detect anomalies in the stream of processed messages. In this context, we will present the special requirements of handling scalable logging systems in highly dynamic infrastructures like enterprise cloud environments, which provide dynamic systems, services and applications