NetAgg: Using Middleboxes for Application-specific On-path Aggregation in Data Centres

Data centre applications for batch processing (e.g. map/reduce frameworks) and online services (e.g. search engines) scale by dis-tributing data and computation across many servers. They typically follow a partition/aggregation pattern: tasks are first partitioned across servers that process data locally, and then those partial re-sults are aggregated. This data aggregation step, however, shifts the performance bottleneck to the network, which typically struggles to support many-to-few, high-bandwidth traffic between servers. Instead of performing data aggregation at edge servers, we show that it can be done more efficiently along network paths. We de-scribe NETAGG, a software platform that supports on-path aggre-gation for network-bound partition/aggregation applications. NET-AGG exploits a middlebox-like design, in which dedicated servers (agg boxes) are connected by high-bandwidth links to network swi-tches. Agg boxes execute aggregation functions provided by ap-plications, which alleviates network hotspots because only a frac-tion of the incoming traffic is forwarded at each hop. NETAGG requires only minimal application changes: it uses shim layers on edge servers to redirect application traffic transparently to the agg boxes. Our experimental results show that NETAGG improves substantially the throughput of two sample applications, the Solr distributed search engine and the Hadoop batch processing frame-work. Its design allows for incremental deployment in existing data centres and incurs only a modest investment cost