LHCb: Universal Logging System for LHCb

In a large-scale IT infrastructure such as the LHCb Online system many applications are running on thousands of machines producing many GBs of logs every day. Although most of the logs are just routine logs, some of them may indicate an attack, a malfunction or provide vital debugging information. Due to their volume only automatisation of the analysis of the logs can provide us with an efficient way to handle all of these logs, ensuring that even the most rare logs will be processed. We present a centralized logging system which allow us to do in-depth analysis of every log. The description of the architecture includes information from how we integrate logging from many devices to a centralized server using syslog and in particular how a correlation can indicate an attack. Special emphasis is given both to security monitoring as well as to the logs that indicate developing malfunctions. To secure our network we have deployed the most known of HIDS, NIDS , LIDS (Host , Network, Log intrusion detection). Each one of them was configured both to cover our needs and communicate with other tools. In some cases , in addition to f configuring the tools, modification to their source code was needed. These modifications are described. Finally we evaluate our work on the performance on live data from our system and show how the predefined requirements are met. We present performance figures, resources needed for the tools and include a comparative study of various tools