Applied Techniques for High Bandwidth Data Transfers across Wide Area Networks

Large distributed systems such as Computational/Data Grids require large amounts of data to be co-located with the computing facilities for processing. Ensuring that the data is there in time for the computation in today’s Internet is a massive problem. From our work developing a scalable distributed network cache, we have gained experience with techniques necessary to achieve high data throughput over high bandwidth Wide Area Networks (WAN). In this paper, we discuss several hardware and software design techniques and issues, and then describe their application to an implementation of an enhanced FTP protocol called GridFTP. We also describe results from two applications using these techniques, which were obtained at the Supercomputing 2000 conference. Large distributed systems such as Computational/Data Grids [10] require large amounts of data to be co-located with the computing facilities for processing. Ensuring that the data is there in time for the computation to start in today’s Internet is a massive problem. At LBNL we developed a high-performance striped data cache called the Distributed Parallel Storage System (DPSS)[29]. The ide