Compilation and Binary Editing for Performance and Security

Traditionally, execution of a program follows a straight and inflexible path starting from source code, extending through a compiled executable file on disk, and culminating in an executable image in memory. This dissertation enables more flexible programs through new compilation mechanisms and binary editing techniques. To assist analysis of functions in binaries, a new compilation mechanism generates data representing control flow graphs of each function. These data allow binary analysis tools to identify the boundaries of basic blocks and the types of edges between them without examining individual instructions. A similar compilation mechanism is used to create individually relocatable basic blocks that can be relocated anywhere in memory at runtime to simplify runtime instrumentation. The concept of generating relocatable program components is also applied at function-level granularity. Through link-time function relocation, unused functions in shared libraries are moved to a section that is not loaded into the memory at runtime, reducing the memory footprint of these shared libraries. Moreover, function relocation is extended to the runtime where functions are continuously moved to random addresses to thwart system intrusion attacks. The techniques presented above result in a 74% reduction in binary parsing times as well as an 85% reduction in memory footprint of the code segment of shared libraries, while simplifying instrumentation of binary code. The techniques also provide a way to make return-oriented programming attacks virtually impossible to succeed