Mitigating Privilege Escalation

One particularly difficult challenge in the computer security landscape is preventing privilege escalation. This type of attack happens when an actor is granted access to some piece of hardware with limited permissions but manages to circumvent the security policies meant to contain them. Although a simple bug in the operating system, or even in user libraries, can be sufficient to enable this type of attack, such a vulnerability is also relatively easy to fix. Privilege escalation mechanisms represent a more challenging security risk because they are methods by which generic vulnerabilities (such as a buffer overflow) can be leveraged to escalate privilege. This thesis describes a collection of operating system hardening techniques designed to mitigate the risks of common privilege escalation mechanisms. This includes non-deterministic loading techniques to randomize code, leveraging the virtualization features of modern hardware to protect operating system code, and a novel operating system design paradigm. A proof-of-concept prototype was developed for each of these techniques using the Bear research microkernel. The code for all techniques described in this thesis is available at https://github.com/SCSLaboratory/BearOS. Each of the techniques described in this thesis is evaluated in terms of the additional security it offers alongside the performance cost of the technique. The security analysis of each technique attempts to describe (and quantify where possible) the types of privilege escalation mechanisms that the technique interrupts. Meanwhile, macro- and micro-benchmarks that are compatible with the Bear microkernel illustrate the practicality of each of these techniques for deployment on real-world systems. Synthesizing four different security mechanisms that each address unique types of privilege escalation threats, the thesis provides a glimpse of a hardened operating system. Contrary to the standard practice of “patching” the status quo in response to each new threat, it attempts to visualize a next-generation operating system design that brings together the best features of non-determinism, virtualization, and hard- ware resource utilization in order to present a more secure computing system that can still meet the ever-increasing performance requirements of modern computing applications