Nonparametric Semi-Supervised Learning for Network Intrusion Detection: Combining Performance Improvements with Realistic In-Situ Training

A barrier to the widespread adoption of learning-based net-work intrusion detection tools is the in-situ training require-ments for effective discrimination of malicious traffic. Su-pervised learning techniques necessitate a quantity of la-beled examples that is often intractable, and at best cost-prohibitive. Recent advances in semi-supervised techniques have demonstrated the ability to generalize knowledge based on a significantly smaller number of provided examples. In network intrusion detection, placing reasonable requirements on the number of training examples provides realistic expec-tations that a learning-based system can be trained in the environment where it will be deployed. This in-situ train-ing is necessary to ensure that the assumptions associated with the learning process hold, and thereby support a rea-sonable belief in the generalization ability of the resulting model. In this paper, we describe the application of a care-fully selected nonparametric, semi-supervised learning algo-rithm to the network intrusion problem, and compare the performance to other model types using feature-based data derived from an operational network. We demonstrate dra-matic performance improvements over supervised learning and anomaly detection in discriminating real, previously un-seen, malicious network traffic while generating an order of magnitude fewer false alerts than any alternative, including a signature IDS tool deployed on the same network