Network-based buffer overflow detection by exploit code analysis

Abstract. Buffer overflow attacks continue to be a major security problem and detecting attacks of this nature is therefore crucial to network security. Signature based network based intrusion detection systems (NIDS) compare network traffic to signatures modelling suspicious or attack traffic to detect network attacks. Since detection is based on pattern matching, a signature modelling the attack must exist for the NIDS to detect it, and it is therefore only capable of detecting known attacks. This paper proposes a method to detect buffer overflow attacks by parsing the payload of network packets in search of shellcode which is the remotely executable component of a buffer overflow attack. By analysing the shellcode it is possible to determine which system calls the exploit uses, and hence the operation of the exploit. Current NIDS-based buffer overflow detection techniques mainly rely upon specific signatures for each new attack. Our approach is able to detect previously unseen buffer overflow attacks, in addition to existing ones, without the need for specific signatures for each new attack. The method has been implemented and tested for buffer overflow attacks on Linux on the Intel x86 architecture using the Snort NIDS