General purpose data streaming platform for log analysis, anomaly detection and security protection

INFN-CNAF is one of the Worldwide LHC Computing Grid (WLCG) Tier-1 data centers, providing computing, networking and storage resources to a wide variety of scientific collaborations, not limited to the four LHC experiments. The INFN-CNAF data center will move to a new location next year. At the same time, the requirements from our experiments and users are becoming increasingly challenging and new scientific communities have started or will soon start exploiting our resources. Currently, we are reengineering several services, in particular our monitoring infrastructure, in order to improve the day-by-day operations and to cope with the increasing complexity of the use cases and with the future expansion of the center. This scenario led us to implement a data streaming infrastructure designed to enable log analysis, anomaly detection, threat hunting, integrity monitoring and incident response. Such a data streaming platform has been organized to manage different kinds of data coming from heterogeneous sources, to support multi-tenancy and to be scalable. Moreover, we will be able to provide an on demand end-to-end data streaming application to those users/communities requesting such a kind of facility. The infrastructure is based on the Apache Kafka platform, which provides streaming of events at large scale with authorization and authentication at the topic level for data isolation and protection. Data can be consumed by different applications, such as OpenSearch, which provides the capability to index a large amount of data and implements appropriate access policies. This poster was presented in the Poster Session 1 of CHEP 2023