Trends, problems and misconceptions on testing Network Intrusion Detection Systems effectiveness

Network Intrusion Detection Systems (NIDS) are hardware or software systems that are used to identify and respond to intrusions in computer networks. An intrusion is a deliberate or accidental unauthorized access to or activity against any of the elements of the network. Evaluation of how effective different intrusion detection technologies are becomes mandatory, in order to know which is the one that better fits in a particular scenario. Nevertheless this is not an easy task. This chapter reviews the main problems regarding testing effectiveness: the absence of standard test methodologies and metrics, the drawbacks of current datasets, the different requirements for testing different technologies, etc. These conditions make evaluation difficult not only for the industry but also for researchers. Scientific proposals are often näively compared. We focus on providing evidence of this situation by means of supporting examples. Some guidelines for the future are finally proposed