Automated Data Analysis for Network Optimization / Threat Detection in Network Architectures

NPS NRP Executive SummaryReport Type: Final ReportProject Summary: The Marine Corps Network Efficiency Lab (MCNEL) is tasked with analyzing very large network traffic archives collected from operations in order to improve future network design, operations, and security. Until this time, MCNEL has used conventional single node packet analyzers, which have proven to be very limiting. Conventional single node packet analyzers are unable to monitor network traffic at scale. In this research, elements of the Apache Hadoop ecosystem, including HBase, Spark, and MapReduce were employed to conduct network traffic analysis on a large collection of network traffic thereby establishing a prototype for network analysis at very large scale in computer clusters. The MCNEL clusters could be organic or in the cloud, perhaps using govCloud cloud computing assets. Initially, limited analysis was conducted directly on packet capture next generation (pcapng) files on the Hadoop Distributed File System (HDFS) using MapReduce. To allow for repeated analysis on the same dataset without reading all source files in their entirety for every calculation, network traffic archives were parsed, and relevant meta- data was bulk loaded into HBase, a Not Only Structured Query Language (NoSQL) database employing the HDFS for parallelization on computer clusters. This NoSQL database was then accessed via Apache Spark where pertinent data is loaded into dataframes, and additional analysis on the network traffic takes place. This research demonstrates the viability of custom, modular, automated analytics, employing open- source software to enable parallelization, to conduct traffic analysis at scale.IMEFMarine Corps Network Efficiency Lab (MCNEL)NPS-18-M034-BApproved for public release; distribution is unlimited