Abstract stack graph to detect obfuscated calls in binaries

Information about calls to the operating system (or kernel libraries) made by a binary executable may be used to determine whether the binary is malicious. Being aware of this approach, malicious programmers hide this information by making such calls without using the call instruction. For instance, the ‘call addr’ instruction may be replaced by two push instructions and a return instruction, the first push pushes the address of instruction after the return instruction, and the second push pushes the address addr. The code may be further obfuscated by spreading the three instructions and by splitting each instruction into multiple instructions. This paper presents a method to statically detect obfuscated calls in binary code. The notion of abstract stack is introduced to associate each element in the stack to the instruction that pushes the element. An abstract stack graph is a concise representation of all abstract stacks at every point in the program. An abstract stack graph, created by abstract interpretation of the binary executables, may be used to detect obfuscated calls and other stack related obfuscations. 1