1On Being Close to the Data: Executing Code in a Replica Management System

Replica management systems have been deployed to improve reliability for the permanent storage of scientific data on cooperative networks of commodity hardware. They also offer the capability of improving the scalability of the data service performance of job schedulers, as executing large batches of data-intensive jobs on a distributed computation system quickly exhausts a centralized storage server. Data sets stored on the compute cluster in a distributed way improve the data access rate, but this technique involves several challenges including logical file name resolution, matchmaking jobs with resources efficiently, and data movement. A variety of relevant distributed computation models that tie replica management systems to job schedulers are described and performance results from a new system are obtained. I