Software protection through obfuscation

Software, over the years, has evolved from free code given along with the hardware for free to a valuable asset, automating almost all of the electronic devices and systems. The growth in the software analyzing tools helped the software developers to analyze and better their software programs. Unfortunately, the same software analyzing tools are used to reverse engineer software systems with malicious intent such as for stealing the intellectual property of the developer, for identifying the vulnerabilities in a program and exploiting them and for unauthorized modifications of the program (tampering). The financial losses incurred by the software industry due to these are in billions. One of the mechanisms to make software reverse engineering harder for an attacker is software obfuscation. Software obfuscation is the process of transforming a program into a semantically equivalent but hard to understand form. The primary objective of our research is to develop software obfuscation algorithms for binary programs so as to make reverse engineering harder for an attacker. In the first part of our research we developed a new software obfuscation algorithm based on self modifying code using stack to conceal the control flow information of binary programs. This will make the reverse engineering of the binary program to assembly level representation harder. In this method, our algorithm translates the control flow instructions, like jump instructions, to normal instructions. The target address of the jump is stored in the stack and the original control flow instructions are reconstructed during runtime by reconstruction instructions. In the next part of our research we proposed a method where encryption and obfuscation are used hand in hand to improve the security of software. In this algorithm, the obfuscation technique used is similar to our previous self modifying code approach. In this method the target addresses are stored in the static data area in an encrypted form. This target addresses are decrypted only during runtime and is re-encrypted after the use. This makes it harder for the attacker to retrieve the target address from the data area. Following the two control flow obfuscation techniques obscuring the control flow within functions, we developed an inter-functional control flow obfuscation technique. One disadvantage of most control flow obfuscation algorithm is that the functions are not affected and the reverse engineering tools can find the beginning and end of a function even after obfuscation. In this method code fragments from each function is stripped from the original function and is stored in another function. Each function will be having code fragments from different functions, thereby creating a function level shuffled version of the original program. Control flow is obscured between and within the function by this method In the last part of our research, we developed and implemented an inter-functional obfuscation based on return instruction. In this method, each function is split into various function blocks, each ending with a return instruction. The function blocks are independent blocks and can be moved within the program, letting the obfuscator shuffle the function blocks, similar to our function level obfuscation technique. A research area of interest, which we can be pursued in the future, is to develop obfuscation algorithm for distributed programs. Devising obfuscation algorithms which take advantage of the features of distributed systems to generate potent obfuscations is a promising future direction. Another research area that can be explored in the future is to use the knowledge of obfuscation to detect obfuscated variants of known malwares. The basic research challenge in this domain is to find features of a program that are invariant to obfuscation.DOCTOR OF PHILOSOPHY (SCE