Malware detection in security operation centres

Malware has evolved from viruses attacking single victims to more sophisticated malware with disruptive purposes. For example, WannaCry ransomware attacks led to hundreds of disruption to NHS care in 2017. Although organizations might have invested in security technologies, their susceptibility to WannaCry hints that the problem goes beyond technology. Security Operations Centres (SOCs) are the first-line of defence in an organisation, providing 24/7 monitoring, detection, and response to security attacks. This thesis aims to explore the challenges in malware detection in Security Operation Centres (SOCs) providing recommendations for possible technological solutions. We first start by investigating the workflow SOC practitioners follow. Through semi-structured interviews, we recognise the analysts' role in the SOC and their interactions with the technological solutions for malware monitoring, detection, investigation and response. Our results highlight the overwhelming reliance on analysts throughout the SOC operations, which might benefit from automation. We elicit the analysts analytical thinking when making decisions, identifying the influential factors that might impact their decision making. Moreover, we investigate security practitioners' perspectives of the security monitoring tools deployed in SOCs and their perception of the high false-positive rates. By identifying the weaknesses and strengths in current SOC tools and challenges in deploying network-monitoring tools, we derive recommendations for future SOC tools development. Understanding the type of malware is an essential step in determining the best response. Sometimes getting access to the infected host is not possible and analysts refer to the network traffic for analysis. Hence, we propose a system that classifies network flow sequences to a malware family. The proposed system is privacy-preserving and effective in classifying a binary to a malware family based on its network traffic, not requiring access to the malware binary itself. Behavioural malware detection approaches are found to be the most reliable by analysts. We propose a behaviour-based malware detection system that improves over state-of-the-art by detecting new or unseen malware. The system uses behavioural high-level network features preserving the privacy of the monitored hosts. Using this system, malware's network activities are captured and modelled as a Markov Chain. Due to the modeling of general bot network behavior by the Markov Chains, the system can detect new malware that has not been seen before making it robust against malware evolution. The novelty of this research is to provide a systematic study on SOCs processes, people, and technology; providing researchers with an understanding of the challenges and opportunities within; bridging that knowledge gap and thereby setting a better foundation for future research in the field.