Security and Protection—Invasive Software

One of the most common forms of security attacks involves exploiting a vulnerability to inject malicious code into an executing application and then cause the injected code to be executed. A theoretically strong approach to defending against any type of code-injection attack is to create and use a process-specific instruction set that is created by a random-ization algorithm. Code injected by an attacker who does not know the randomization key will be invalid for the random-ized processor effectively thwarting the attack. This paper describes a secure and efficient implementation of instruc-tion-set randomization (ISR) using software dynamic transla-tion. The paper makes three contributions beyond previous work on ISR. First, we describe an implementation that uses a strong cipher algorithm—the Advanced Encryption Stan-dard (AES), to perform randomization. AES is generally believed to be impervious to known attack methodologies. Second, we demonstrate that ISR using AES can be imple-mented practically and efficiently (considering both execu-tion time and code size overheads) without requiring special hardware support. The third contribution is that our approach detects malicious code before it is executed. Previ-ous approaches relied on probabilistic arguments that execu-tion of non-randomized foreign code would eventually cause a fault or runtime exception