Data-driven Approach to Information Sharing using Data Fusion and Machine Learning

The number of security incidents worldwide is increasing, and the capabilities to detect and react is of uttermost importance. Intrusion Detection Systems (IDSs) are employed in various locations in networks to identify malicious activity. These sensors produce large amounts of data, which are fused and reduced. It is necessary to determine how to perform such fusion and reduction of data from heterogeneous sources. IDS is known to produce a high amount of false positives which create a high workload for human analysts at Security Operation Center (SOC). To ensure scalability, systems for reducing and streamlining the detection process is critical. The application of Threat Intelligence (TI) in information security for detection and prevention is widespread. When performing sharing of TI, it must be ensured that the data is reliable and trustworthy. Further, it must be guaranteed that the sharing process leaks sensitive data. This thesis has proposed a process model describing the process of fusion and reduction of heterogeneous sensor data and TI in intrusion detection. Our work is based on a literature study and qualitative research interviews with security experts from law enforcement and public and private organisations. Further, an identification of reliable and trustworthy features in such fused and reduced data for use in Machine Learning (ML) is given. We have applied data-driven methods on a real-world dataset from a SOC for this identification, and evaluate our results using well-known performance measure. Our results show that the application of ML can be used for prediction and decision support in the operation of SOC. We also provide an identification of sensitive features from the features selected by our data-driven experiments