Re)Design considerations for scalable large-file content distribution

The CoBlitz system was designed to provide efficient large file transfer in a managed infrastructure environ-ment. It uses a content distribution network (CDN) cou-pled with a swarm-style chunk distribution system to re-duce the bandwidth required at origin servers. With 6 months of operation, we have been able to observe its be-havior in typical usage, and glean information on how it could be redesigned to better suit its target audience. At its heart, this paper describes what happens when a plausible conceptual design meets the harsh realities of life on the Internet. We describe our experiences improv-ing CoBlitz’s performance via a range of techniques, in-cluding measurement-based feedback, heuristic changes, and new algorithms. In the process, we triple CoBlitz’s performance, and we reduce the load it places on origin servers by a factor of five. In addition to improving per-formance for CoBlitz’s users, we believe that our experi-ences will also be beneficial to other researchers working on large-file transfer and content distribution networks.