Using Regression Techniques to Predict Large Data Transfers

The recent proliferation of Data Grids and the increas-ingly common practice of using resources as distributed data stores provide a convenient environment for com-munities of researchers to share, replicate, and manage access to copies of large datasets. This has led to the question of which replica can be accessed most effi-ciently. In such environments, fetching data from one of the several replica locations requires accurate predic-tions of end-to-end transfer times. The answer to this question can depend on many factors, including physi-cal characteristics of the resources and the load behav-ior on the CPUs, networks, and storage devices that are part of the end-to-end data path linking possible sources and sinks. Our approach combines end-to-end application through-put observations with network and disk load variations and captures whole-system performance and variations in load patterns. Our predictions characterize the effect of load variations of several shared devices (network and disk) on file transfer times. We develop a suite of univariate and multivariate predictors that can use multi-ple data sources to improve the accuracy of the predic-tions as well as address Data Grid variations (availability of data and sporadic nature of transfers). We ran a large set of data transfer experiments using GridFTP and ob-served performance predictions within 15 % error for our testbed sites, which is quite promising for a pragmatic system. Key words: Grids, data transfer prediction, replica se-lection.