Behaviour-based malware detector

The number of new malwares created every day is at an all-time high, one of the main reasons is that malware authors are using polymorphic methods. As a result, antivirus software is not able to detect the malware in a timely manner, therefore, it is insufficient to use an antivirus software to detect new malware. Over the last few years, machine learning techniques have shown to be able to have high accuracy in classification tasks. Anti-malware solutions have also started using machine learning techniques to classify malware. This report presents a proof of concept software that uses convolutional neural network (CNN) to perform malware classification. Firstly, malicious and non-malicious files were collected and analysis was done for each of the files using a static analysis tool, pestudio. Subsequently, the important features were extracted and used to train the CNN to perform malware classification. The neural network is able to achieve 74% classification accuracy, with 80% of the malicious files detected. Moreover, support for detecting malware in PDF, Microsoft Office and Adobe Flash files is needed due to the increasing number of malware which targets these file formats. The open source tools: PeePDF, Olevba and SWFDump were used to help to detect malware in PDF, Microsoft Office and Adobe Flash files respectively. Last but not least, testing for sandbox evasion techniques is necessary due to the increasing number of malwares that are able to detect the presence of a sandbox. To prevent malware from detecting the sandbox environment, the sandbox environment has to be tested first using a benign malware from the open source tool, Al-Khaser, before analysing the malware using that sandbox. This ensures that a malware would not be employing any of the commonly used evasion techniques tested in the sandbox.Bachelor of Engineering (Computer Science