Measuring Intrusion Detection Capability: An Informmation-Theoretic Approac

A fundamental problem in intrusion detection is what met-ric(s) can be used to objectively evaluate an intrusion detec-tion system (IDS) in terms of its ability to correctly classify events as normal or intrusive. Traditional metrics (e.g., true positive rate and false positive rate) measure different as-pects, but no single metric seems sufficient to measure the capability of intrusion detection systems. The lack of a sin-gle unified metric makes it difficult to fine-tune and evaluate an IDS. In this paper, we provide an in-depth analysis of ex-isting metrics. Specifically, we analyze a typical cost-based scheme [6], and demonstrate that this approach is very con-fusing and ineffective when the cost factor is not carefully se-lected. In addition, we provide a novel information-theoretic analysis of IDS and propose a new metric that highly com-plements cost-based analysis. When examining the intru-sion detection process from an information-theoretic point of view, intuitively, we should have less uncertainty about the input (event data) given the IDS output (alarm data). Thus, our new metric, CID (Intrusion Detection Capability), is defined as the ratio of the mutual information between the IDS input and output to the entropy of the input. CID has the desired property that: (1) It takes into account all the important aspects of detection capability naturally, i.e., true positive rate, false positive rate, positive predictive value, negative predictive value, and base rate; (2) it objectively provides an intrinsic measure of intrusion detection capa-bility; and (3) it is sensitive to IDS operation parameters such as true positive rate and false positive rate, which can demonstrate the effect of the subtle changes of intrusion de-tection systems. We propose CID as an appropriate perfor-mance measure to maximize when fine-tuning an IDS. The obtained operation point is the best that can be achieved by the IDS in terms of its intrinsic ability to classify input data. We use numerical examples as well as experiments of actual IDSs on various data sets to show that by using CID