Precise and Efficient Dynamic Analysis of Systems Software

Today's mainstream operating systems (OSs) have monolithic kernels, in which low-level systems software such as device drivers, networking systems, and file systems all run within the kernel with no separation of privilege between them. This means that exploiting a single vulnerability present in any kernel subsystem gives adversaries access to the entire OS. It has been repeatedly demonstrated that OS kernels can be compromised locally or even remotely, through their wide attack surface—the system call interface as well as the peripheral interface.There exists a significant body of research aimed at finding vulnerabilities in software. Fuzzing, among others, is widely regarded as a practical and effective approach to finding vulnerabilities. Most of the fuzzing research, however, has targeted user-space software. Unfortunately, fuzzing systems software running in kernel space can be more challenging than fuzzing user-space software, as existing kernel fuzzers suffer from unique challenges that arise in kernel space. More specifically, existing fuzzers for OS kernels are (i) imprecise in that they do not accurately model the full capabilities of possible attackers, e.g., attackers on the peripheral hardware side, or (ii) inefficient due to various delays caused by system crashes, asynchronous input processing, etc. This dissertation presents two dynamic analysis techniques that significantly alleviate these problems: (i) a technique that enables fuzzing the peripheral input space of OS kernels, which precisely models the capabilities of a strong attacker on the peripheral side, and (ii) a virtual machine checkpointing technique that can accelerate OS kernel fuzzing, making dynamic analysis more efficient. The dissertation concludes with a summary of these techniques as well as a recommendation for promising future work directions