Analysis of Evasion Techniques in Web-based Malware

Web-based mechanisms, often mediated by malicious JavaScript code, play an important role in malware delivery today, making defenses against web-based malware crucial for system security. To make it even more challenging, malware authors often take advantage of various evasion techniques to evade detection. As a result, a constant arms race of evasion and detection techniques between malware authors and security analysts has led to advancement in code obfuscation and anti-analysis techniques. This dissertation focuses on the defenses against web-based malware protected by advanced evasion techniques from both defensive and offensive perspectives. From a defensive perspective, we examine existing evasion techniques and propose deobfuscation and detection approaches to defeating some popular techniques used by web-based malware today. In the case of code-unfolding based obfuscation, we use a semantics-based approach to simplify away obfuscations by identifying code that is relevant to the behavior of the original program. In the case of environment-dependent malware, we propose environmental predicate, which detects behavior discrepancy of JavaScript program between targeted browser and detector sandbox, therefore protecting users from possible detection false negatives caused by environmental triggers. From an offensive perspective, we analyze existing detection techniques to examining their assumptions and study how these assumptions can be broken. We also propose a combination of obfuscation and anti-analysis techniques, targeting these limitations, which can hide existing web-based malware from state-of-the-art detectors