Hybrid Analysis of Executables to Detect Security Vulnerabilities

Abstract—Detection of vulnerabilities in executables is one of the major problems facing the software industry. Cracking has increased the complexity where even the programs from reputed companies might be made malicious if they are vulnerable. The main challenge in analysis of executables is due to the unavailabil-ity of the source code. Results generated using dynamic analysis alone are unsound, i.e., they do no generalize. On the other hand, results generated using static analysis are usually conservative and it will report weaker properties which might be true but useless. Also, using a model in static analysis which is complete, will render the analysis complex and slow. In this work, we present a hybrid approach, which uses a combination of static and dynamic analysis to identify vulnerabilities. In this approach, we first instrument the executable to extract the control flow of the program. We get the assembly code of the instructions being executed. We then perform static analysis on the assembly code using control flow and register bounds. Static analysis uses slicing on the disassembled code and constraint bound checking is performed on the slice generated. In this way, we exploit the synergy between static and dynamic analysis. Memory errors including memory leaks, buffer overflow and dangling pointers found during the hybrid analysis are reported. Index Terms—Hybrid analysis, security vulnerabilities, mali-cious code, memory errors, instrumentation, slicing, validation. I