A Framework for Collection and Management of Intrusion Detection Data Sets

Two areas in intrusion detection research receive little attention: data collection and data management. Gigabit Ethernet is becoming widely deployed, with ten gigabit Ethernet not far behind. Many current solutions strain under such bandwidth rates, resulting in data loss. This is unacceptable for accurate, reliable intrusion detection systems. Data management solutions vary greatly from product to product. Typically, older data is periodically migrated to some archived format. Once archived, the data set cannot be easily queried or analyzed without being imported back into the original tool. This makes forensics and trend analysis extremely difficult. This paper addresses data collection and management for intrusion detection by providing a framework designed to accommodate high-volume, heterogeneous data sets. This framework solves many of the problems of conventional approaches to intrusion detection. Distributed computing is leveraged to assure scalability. Data can be captured, queried and analyzed in real-time; data set sizes are limited only by available storage. Benchmarks of the initial prototype are also provided