Design for Security: Measurement, Analysis and Mitigation Techniques

Security vulnerabilities pose a serious threat to computer systems and network infrastructures. This dissertation addresses the measurement and analysis of security vulnerabilities and their impact, as well as the design of several techniques for vulnerability mitigation. The research starts with the analysis of the security vulnerabilities published in the Bugtraq list and CERT advisories. An in-depth analysis of vulnerability reports and the corresponding source code of the applications motivates our development of a finite state machine (FSM) model to reason about security vulnerabilities. Most attacks can be decomposed into a series of violations of simple predicates, which provides a more formal way to depict these attacks. Besides the analysis of security vulnerabilities, many real-world security attacks are analyzed. The analysis shows that, although most current memory-corruption-based attacks compromise system security by overwriting control data, a new type of attack, namely non-control-data attack, can also be generally applicable to real-world software, e.g., they break into network servers to obtain the root privilege. This emphasizes the necessity of further research on defenses against memory-corruption-based attacks, including control-data attacks and non-control-data attacks. The notion of pointer taintedness is introduced as the basis for detecting memory-corruption-based attacks. A pointer is said to be tainted if its value comes directly or indirectly from user input. Pointer taintedness allows the user to arbitrarily specify the target memory address to read, write, or transfer control to, which is usually a pathological program behavior. On the other hand, the attacker.s ability to taint a pointer value is a crucial requirement for most attacks. Based on the notion of pointer taintedness, a theorem-proving technique is developed to identify potential security vulnerabilities via static source code analysis, and a processor architecture mechanism is implemented for dynamic pointer taintedness detection. Evaluations show that the proposed techniques offer better security coverage than existing methods: by detecting pointer taintedness, both control-data and non-control-data attacks are defeated in a unified manner