Detecting unknown anomalous program behavior using API system calls

This paper presents the detection techniques of anomalous programs based on the analysis of their system call traces. We collect the API calls for the tested executable programs from Microsoft detour system and extract the features for our classification task using the previously established n-gram technique. We propose three different feature extraction approaches in this paper. These are frequency-based, time-based and a hybrid approach which actually combines the first two approaches. We use the well-known classifier algorithms in our experiments using WEKA interface to classify the malicious programs from the benign programs. Our empirical evidence demonstrates that the proposed feature extraction approaches can detect malicious programs over 88% which is quite promising for the contemporary similar research.