Detection of Worm Propagation Engines in the System Call Domain using Colored Petri Nets

While network worms are different in payloads and utilized exploits, they all have one common component, the propagation engine. It is important that the number of conceptually distinctive propagation engines employed by existing network worms is quite limited. This paper presents a novel approach for detecting attacks perpetrated by network worms. It implies the recognition of the propagation engine functionality in the process behavior on the system call level and attributing it to the shell code activity being the first stage in the worm proliferation. We suggest an elegant way to trace the behavior of a computer process in the system call domain and assess its current functionality through the utilization of Colored Petri Nets. We developed, tested and evaluated a Propagation Engine Detector (PED) system which detects the worm shell code activity performed by a process during the attack. Moreover, PED recognizes the type of propagation engine employed by the attacking worm