Using Instruction Sequence Abstraction for Shellcode Detection and Attribution

Abstract—Although several research teams have focused on bi-nary code injection, it is still an unsolved problem. Misuse-based detection lacks the flexibility to tackle unseen malicious code samples and anomaly-based detection on byte patterns is highly vulnerable to byte cramming and blending attacks. In addition, it is desperately needed to correlate newly-detected code injection instances with known samples for better understanding the attack events and tactically mitigating future threats. In this paper, we propose a technique for modeling shellcode detection and attribu-tion through a novel feature extraction method, called instruction sequence abstraction, that extracts coarse-grained features from an instruction sequence. Our technique facilitates a Markov-chain-based model for shellcode detection and support vector machines for encoded shellcode attribution. We also describe our experimental results on shellcode samples to demonstrate the effectiveness of our approach. I