CipherXRay: Exposing Cryptographic Operations and Transient Secrets from Monitored Binary Execution

Abstract—Malwares are becoming increasingly stealthy, more and more malwares are using cryptographic algorithms (e.g., packing, encrypting C&C communication) to protect themselves from being analyzed. The use of cryptographic algorithms and truly transient cryptographic secrets inside the malware binary imposes a key obstacle to effective malware analysis and defense. To enable more effective malware analysis, forensics and reverse engineering, we have developed CipherXRay – a novel binary analysis framework that can automatically identify and recover the cryptographic operations and transient secrets from the execution of potentially obfuscated binary executables. Based on the avalanche effect of cryptographic functions, CipherXRay is able to accurately pinpoint the boundary of cryptographic operation and recover truly transient cryptographic secrets that only exist in memory for one instant in between multi-ple nested cryptographic operations. CipherXRay can further identify certain operation modes (e.g., ECB, CBC, CFB) of the identified block cipher and tell whether the identified block cipher operation is encryption or decryption in certain cases. We have empirically validated CipherXRay with OpenSSL, popular password safe KeePassX, the ciphers used by malware Stuxnet, Kraken and Agobot, and a number of third party softwares with built-in compression and checksum. CipherXRay is able to identify various cryptographic operations and recover cryptographic secrets that exist in memory for only a few microseconds. Our results demonstrate that current software implementations of cryptographic algorithms hardly achieve any secrecy if their execution can be monitored. Index Terms—Binary analysis, avalanche effect, key recovery attack on cryptosystem, transient cryptographic secret recovery, secrecy of monitored execution, reverse engineering. I