Enabling high data throughput in desktop grids through decentralized data and metadata management: The blobseer approach

Abstract. Whereas traditional Desktop Grids rely on centralized servers for data management, some recent progress has been made to enable distributed, large in-put data, using to peer-to-peer (P2P) protocols and Content Distribution Networks (CDN). We make a step further and propose a generic, yet efficient data storage which enables the use of Desktop Grids for applications with high output data re-quirements, where the access grain and the access patterns may be random. Our solution builds on a blob management service enabling a large number of con-current clients to efficiently read/write and append huge data that are fragmented and distributed at a large scale. Scalability under heavy concurrency is achieved thanks to an original metadata scheme using a distributed segment tree built on top of a Distributed Hash Table (DHT). The proposed approach has been imple-mented and its benefits have successfully been demonstrated within our BlobSeer prototype on the Grid’5000 testbed.