Hardware-assisted online defense against malware and exploits

Security is a major concern in the computing systems with the increasing number of cyber attacks in recent years. Mainstream security solutions (e.g., antivirus, scanners) are mostly implemented as software. Thus, the vulnerabilities in these solutions can be exploited to disable or bypass the defense, like rootkit and bootkit. Besides, software approaches suffer from the high performance overhead and resource requirement. As the result, they can only be implemented partially, which leaves opportunities for the adversaries to exploit the system. Recently, hardware-assisted solutions for cyber security have emerged as a promising protection against the evolving attacks. Compared to the software solutions, hardware-based solutions have several advantages. First, they are difficult to be bypassed or identified by malware as they are running below the operating system. Second, hardware supported security approaches are much more energy- and power-efficient, which are desirable for runtime defense. Third, hardware-based implementation offers unmatched visibility into the program execution, which provides opportunities to develop novel security techniques. Finally, the implementation in hardware inherently offers the quick detection of attacks. It runs separately from the processor and hence, barely affects the processor performance. In this thesis, we study hardware-assisted security approaches to defend against cyber attacks at runtime. We focus on two critical issues of software security: malware and exploits. Malware (short for “malicious software”) is a collective term for any program that enters into a system without the knowledge of the user and deliberately fulfills the harmful intent of an attacker. An exploit refers to a software program that attacks the system by taking advantage of a vulnerability present in the system. Adversary commonly uses exploits to attack an operating system or application vulnerability to gain privileges, so that they can run malicious code in the system. Hence, the defense at the exploitation stage can eliminate the threats at the point of pre-infection and protect from malware download and its execution. We begin with analyzing malware and proposing detection solutions. 1. First, we propose a hardware-enhanced architecture to detect malware at runtime. Our approach aims to capture the malicious features (i.e., high-level semantics) of malware. We develop a machine learning approach in FPGA to train a classifier using features obtained from the known samples. At runtime, the trained classifier is used to classify the unknown samples as malware or benign, with early prediction. The proposed machine learning approach can effectively detect unknown malware samples based on the trained features and this approach is highly scalable. There are two challenges for this approach: 1) it can be defeated if features in the machine learning method are not comprehensive or if malware uses a new attacking strategy; and 2) the machine learning approach cannot explain the attack behavior of malware and hence fails for malware classification. 2. Second, we propose a semantic-based malware detection method to address the two challenges above. We firstly propose to use deterministic finite automata (DFA) to model the malware behavior. Then we learn the attack model of malware via an offline analysis. At runtime, we implement a DFA-based detection approach in hardware to check whether a program’s execution contains the malicious behavior specified in the DFA. This approach can effectively capture the attack behavior of malware and has the potential to detect zero-day malware. Implemented in hardware, our architecture offers a real-time detection with low performance and resource overhead. More importantly, it cannot be bypassed by malware using sophisticated evasive techniques. To have a fine-grained understanding of malware, we investigate the common exploitation techniques in the second half of the thesis. 1. First, we present a fine-grained control flow integrity approach to defend against runtime memory attacks using a hardware-enhanced architecture. Based on the offline profiling of the original benign program, our security model can effectively detect memory attacks (injected in the original program) at runtime, which works in parallel to the processor with low performance (