A Hybrid Framework for the Systematic Detection of Software Security Vulnerabilities in Source Code

In this thesis, we address the problem of detecting vulnerabilities in software where the source code is available, such as free-and-open-source software. In this, we rely on the use of security testing. Either static or dynamic analysis can be used for security testing approaches, yet both analyses have their advantages and drawbacks. In fact, while these analyses are different, they are complementary to each other in many ways. Consequently, approaches that would combine these analyses have the potential of becoming very advantageous to security testing and vulnerability detection. This has motivated the work presented in this thesis. For the purpose of security testing, security analysts need to specify the security properties that they wish to test software against for security violations. Accordingly, we firstly propose a security model called Team Edit Automata (TEA), which extends security automata. Using TEA, security analysts are capable of precisely specifying the security properties under concerns. Since various code instrumentations are needed at different program points for the purpose of profiling the software behavior at run-time, we secondly propose a code instrumentation profiler. Furthermore, we provide an extension to the GCC compiler to enable such instrumentations. The profiler is based on the pointcut model of Aspect-Oriented Programming (AOP) languages and accordingly it is capable of providing a large set of instrumentation capabilities to the analysts. We particularly explore the capabilities and the current limitations of AOP languages as tools for security testing code instrumentation, and propose extensions to these languages to allow them to be used for such purposes. Thirdly, we explore the potential of static analysis for vulnerability detection and illustrate its applicability and limitations. Fourthly, we propose a framework that reduces security vulnerability detection to a reachability problem. The framework combines three main techniques: static analysis, program slicing, and reachability analysis. This framework mainly targets software applications that are generally categorized as being safety/security critical, and are of relatively small sizes, such as embedded software. Finally, we propose a more comprehensive security testing and test-data generation framework that provides further advantages over the proposed reachability model. This framework combines the power of static and dynamic analyses, and is used to generate concrete data, with which the existence of a vulnerability is proven beyond doubt, hence mitigating major drawbacks of static analysis, namely false positives. We also illustrate the feasibility of the elaborated frameworks by developing case studies for test-data generation and vulnerability detection on various-size software