Using system call analysis to stop evasion attacks in anomaly based Intrusion Detection System

Intrusion Detection Systems (IDSs) that operate on the principle of system call monitoring are known to be susceptible to mimicry or evasion attacks. It has been shown that an intelligent adversary armed with comprehensive knowledge of the target system or network, can penetrate these targets, hide his presence from the IDS, and continue to carry out damage. IDSs, which use system calls to define normal behavior, often leave out complimentary information about them, and intruders use precisely this drawback, to deceive the IDS. This thesis investigates the vulnerabilities of a system call based IDS and carries out a theoretical and experimental study of methods allowing to improve the IDS performance and reliability. It analyzes the design principles and architecture of anomaly based IDSs and studies the implementation of a typical system call based anomaly IDS. This category of anomaly detection systems is currently attracting considerable attention within the research community and various prototypes have been developed in recent years. The thesis investigates the hypothesis that by monitoring the number of system calls that fail and return error values on a per process basis, it would be possible to identify abnormally behaving processes. It also suggests that by using only a certain set of critical system calls instead of all the defined calls, it could be possible to detect and stop mimicry attacks. pH IDS is used for the purpose of the experiments as its source code is freely available. It works as a patch to the Linux kernel and alters the way system calls are handled. The tests were carried out on a stand-alone Linux box running RedHat 9 with kernel version 2.4.20. Local exploits, which were readily available on the Internet, were used in the experiments. Some of the results obtained contradicted our original hypothesis and are indicative of the scope for future work in this area. The tests revealed that it was not possible to simply use system call return values to identify erroneously behaving processes. However after classifying the system calls into critical and non-critical sets, a form of mimicry attacks could be successfully detected. The results confirm the potential of this technique to thwart evasion attacks and points to the direction of possible further work in this area