Security Engineering of Hardware-Software Interfaces

Hardware and software do not operate in isolation. Neither should they be regarded as such when securing systems. To seamlessly facilitate computing, they have to communicate via interfaces. Besides characterizing the means by which software can harness the exposed functionalities of hardware, these hardware-software interfaces define the degree and granularity of control and access that software possesses to the lower layers of the system stack. These mechanisms provide a rich source of hardware assistive technologies that can be tapped to enhance security as a full-system property. On the flip side, given the level of access software has to these hardware features, security-oblivious designs of hardware and their interfaces can expose systems to new vulnerabilities. Evidently, these hardware-software interfaces represent a crucial focal area in systems for the formulation, review and refinement of security measures. This dissertation advances the thesis that security as a full-system property can be improved by examining and leveraging the interworking of hardware and software. It advocates a full-system approach in architecture design by demonstrating how unanticipated ways in which hardware and software co-operate can induce unintended computing behavior and pose security risks. It develops novel techniques to repurpose commodity hardware support to create new defense primitives that exploit the synergy between hardware and software. It shows how commodity hardware-software interfaces play an instrumental role in security with the hardware's well-positioned access to runtime information. All these interface-oriented design principles, as this dissertation demonstrates, are widely applicable and practical as the highlighted three case studies span the three primary stages of a typical security attack, namely the act of inducing unintended system behavior, exploiting vulnerability to achieve initial system control, and executing malicious code for nefarious goals. First, the dissertation begins by scrutinizing the design of energy management mechanisms, a prevalent class of hardware-software interfaces found in almost all commodity systems. It shows, for the first time, that as we pursue increasingly aggressive cooperative hardware-software mechanisms to improve energy efficiency, doing so with no regard for security can create serious vulnerabilities. This dissertation highlights a multitude of issues in the current designs of energy management mechanisms. It further demonstrates how, with fine-grained software-based control of underlying hardware voltage and frequency regulators, attackers can exploit these issues to induce unintended computing behavior. It shows that beyond causing unintended system behavior, abusing these interfaces in security-oblivious energy management designs can violate all three key security properties in spite of hardware-enforced isolation: confidentiality (extracting AES keys), integrity (loading self-signed code), and clearly, availability (freezing the device). Second, the dissertation addresses an advanced class of dynamic code reuse exploits that rely on memory disclosure vulnerabilities to construct their initial payload code at runtime. This class of exploits bypasses even the finest-grained randomization-based defenses. While the concept of execute-only memory in existing defenses works well, it cannot be applied effectively in closed-source systems where perfect disassembly of compiled binaries is not possible. To tackle this problem, this dissertation first introduces the Destructive Code Read primitive---a defense technique that randomizes executable memory as it is being read as data---as a means to thwart memory disclosure exploits as well as to sidestep the problem of imperfect binary disassembly in COTS systems. It leverages the virtualization assistive hardware feature to timely mediate read operations into executable memory, thereby significantly lowering the cost of deploying the Destructive Code Read defense primitive. Tapping into the unique strengths of functionality closer to the hardware layer of the system stack, it extends the benefits of execute-only memory defenses to COTS systems. Finally, the dissertation builds on the insight that hardware, being the lowest part of the system stack, is uniquely positioned to augment traditionally software-only techniques. Besides being more performant and energy-efficient, hardware offers extensive visibility into code execution at the software layers. This dissertation shows that these hardware characteristics offer unprecedented insights into code execution, both benign and malicious. It demonstrates that the interaction of hardware and software can be modeled as microarchitectural events, which can in turn be leveraged to detect anomalous malicious code execution in the latter stages of a security attack. Using assistive debugging hardware features to efficiently audit these events, it further develops novel techniques to make sense of the noisy and lower-level microarchitectural events to detect in-flight shellcode execution and full-fledged anomalous malicious programs