Improving the Effectiveness of Behaviour-based Malware Detection

Malware is software code that has malicious intent. In recent years, there have beenhuge changes in the threat landscape. As our dependency on the Internet for social related information sharing and work increases, the number of the possible threats ishuge and we are indeed susceptible to them. Attacks may from the individual ororganisational level, to nation-states resorting to cyber warfare to infiltrate and sabotageenemies operation. Hence, the need for a secure and dependable cyber defence isrelevant at all levels.Malware can only do harm if it is allowed to propagate and execute without beingdetected. Detection based on signature alone is not the answer, because new malwarewith new signatures cannot be detected. Thus, behaviour-based detection is needed todetect novel malware attacks. Moreover, malware detection is a challenging task whenmost of the latest malware employs some protection and evasion techniques. In thisstudy, we present a malware detection system that addresses both propagation andexecution. Detection is based on monitoring session traffic for propagation, and API callsequences for execution. Our approach is inspired by the human immune systemtheories known as the Self/Non-self Theory and the Danger Theory.For malware detection during propagation, we investigate the effectiveness ofsignature-based detection, anomaly-based detection and the combination of both. Thedecision-making relies upon a collection of recent signatures of session-based trafficdata collected at the endpoint (single computer) level. Patterns in terms of portdistributions and frequency or session rates of the signatures are observed. If anabnormality is found, it often signifies worm behaviour. A knowledge base consisting ofrecent traffic data, which is used to predict future traffic patterns, helps to reverse theincorrect flagging of suspected worms. The knowledge base is made of recent traffic,used to predict future patterns of traffic data. It maintains only recent data as the usagepattern of a computer changes over time.Our proposed system includes several detectors, the operations of which aregoverned by several parameters. We study both how these parameters affect the resultsand performances when different detectors are or are not included. We find that thedetectors produce inconsistent results when used independently but when used together achieve promising detection rates. In addition, we identify which worms areconsistently detected by the system, and the characteristics of those the system cannotdetect well.For detection based on execution, we analyse sequences of API calls grouped into ngrams which are compared with benign and malware profiles. A decision is made based on a statistical measure, which indicates how close the behaviour represented in the ngrams is to each of the profiles. Experiments show that the system is capable ofcorrectly detecting malware early in its execution.The main contributions of this thesis are: the proposal and evaluation of aframework for detecting malware, that considers both propagation and execution in asystematic way; the detection methods are based on information that is simpler toprocess than other proposals in the literature, yet still achieve very high detectionaccuracy; and malware can be correctly recognised early in its execution.The experimental results show that our framework is promising in terms of effectivebehaviour-based detection that can detect malware and protect our computer networksfrom future zero-day attacks