Compiler-Based Mitigations of Vulnerabilities in Systems Software

Systems software written in C/C++ is plagued by bugs, which attackers exploit to gain control of systems, leak sensitive data, or perform denial-of-service attacks. This plethora of vulnerabilities is caused by C/C++ not enforcing memory or type safety in language by design, instead they leave security checks to the programmer. Previous research primarily focuses on preventing control-flow hijack attacks. In a control-flow hijack attack, the attacker manipulates a return address or function pointer to cause code of her choosing to be executed. Abadi et al. propose Control- Flow Integrity (CFI), to prevent such attacks, but as our CFI survey shows, CFI mechanisms have varying degrees of precision. Researchers exploit the imprecision in CFI implementations to evade their protection. One area of imprecision in CFI mechanisms is virtual functions in C++ programs. Attackers can re-target virtual function calls to other invalid functions as part of an exploit. Our work, VTrust, provides specialized protection for C++ virtual functions with low overhead. As CFI mechanisms improve, and are widely deployed, attackers will follow the path of least resistance towards other attack vectors, e.g., non-control-data attacks. In a non-control-data attack the attacker manipulates ordinary variables (not return addresses, function pointers, etc.) to carry out the attack. Non-control-data attacks are not prevented by CFI, because the control-flow follows a valid path in the original program. The attack is carried out by modifying only non-control-data. To address this emerging problem, we have developed Data Confidentiality and Integrity (DCI) which allows the programmer to select which data types should be protected from corruption and information leakage by the attacker. In this dissertation, we propose that by using static analysis and runtime checks, we can prevent attacks targeted at sensitive data with low overhead. We have evaluated our techniques, VTrust and DCI, on the SPEC CPU2006 benchmarks, the Firefox web browser, and the mbedTLS cryptographic library. Our results show our implementations have lower performance overhead than other state-of-the-art mechanisms. In our security evaluation, we have several case studies which show our defenses mitigate publicly disclosed vulnerabilities in widely deployed software. In future work, we plan to improve our static sensitivity analysis for DCI and investigate new methods for automatically identifying sensitive data