Adapting Block-Sized Captures for Faster Network Flow Analysis on the Hadoop Ecosystem

With the rapid and continuous growth of annual network traffic comes the need to develop systems that can efficiently scale to meet the demands of analyzing all this traffic data. The Hadoop ecosystem provides an environment that is capable of addressing this need, because of its horizontal scalability and its data locality optimization feature. The latter feature improves parallel analysis of data by placing computing tasks within the same node that contains the block of data to be analyzed. However, this feature cannot be taken advantage of by those input formats that are not splittable within the Hadoop Distributed File System. The PCAP format used for capturing network data is one such file format. To address this issue, this paper proposes the inclusion of a minimal preprocessing step before PCAP files are fed into Hadoop and analyzed using the hcap framework, which is currently the fastest framework for analyzing PCAP data in Hadoop. This preprocessing step is designed to adapt the PCAP files into properly split blocks in order to take advantage of Hadoop\u27s data locality optimization feature. Results show a significant improvement in query response time with a performance gain of 92%, 89%, 91%, and, 87% for scan, aggregate, join, and aggregate-join queries respectively when compared to the original hcap framework