System memory protection and vulnerability assessment in presence of software attacks

Software vulnerabilities widely exist among various software from operating system kernel to web browser, from PC to embedded device. The arms race is continuing between new vulnerability exploit techniques and new mitigations. The essential part of protecting software from compromising relies on system memory protection in specific ways. Addressing protection of system critical variables, heap layout, and user variables that are referenced freely from the kernel are the state-of-art challenges. This dissertation aims at protecting the above-mentioned vulnerabilities that exist in the wild and presents systematic mitigation solutions. For each specific vulnerability, our mitigation either leverages a new CPU features such as Intel SGX or an existing CPU feature in a novel way to achieve adequate protection with a modest performance overhead. Additionally, we utilize a software-only method to solve the use-after-free vulnerability in the web browsers — a trade-off between the deterministic heap layout and memory usage. Furthermore, we develop a new software attack that is parasitic on an extra piece of hardware circuit to assess conventional software mitigations’ effectiveness. We evaluated each system with real-world vulnerabilities and their exploits that are publicly available. The results show that these mitigations can effectively protect the system with an acceptable performance overhead. Our parasitic-hardware-based attack reveals the possibility of being deployed in the field devices such critical controllers (e.g., programmable logic controllers PLCs) in cyber-physical platforms such as the power grid infrastructures. This type of attack completely evades conventional software mitigation techniques.Ph.D.Includes bibliographical reference