File Chunks Balancing For DFSs by Load Rebalancing Algorithm

Abstract—Distributed File Systems are file systems that allow access to files from multiple hosts via a computer network, making it possible for multiple users on multiple machines to share files and storage resources. These are the key building blocks for cloud computing applications based on the Map Reduce programming paradigm. In a cloud computing environment, failure is the norm, and chunk servers may be upgraded, replaced, and added in the system. In addition, files can also be dynamically created, deleted and appended. And that lead to load imbalance in a distributed file system. It means that file chunks are not distributed equitably between the nodes. Emerging distributed file systems in production systems strongly depend on a central node for chunk reallocation. However, this centralized approach can provoke a bottleneck for those central nodes as they become unable to manage a large number of file accesses. Consequently, dealing with the load imbalance problem with the central nodes complicate more the situation as it increases their heavy loads. A fully distributed load rebalancing algorithm is presented here to cope with the load imbalance problem. Additionally, aspires to reduce network traffic or movement cost caused by rebalancing the loads of nodes as much as possible to maximize the network bandwidth available to normal applications. Moreover, as failure is the norm, nodes are newly added to sustain the overall system performance resulting in the heterogeneity of nodes. Exploiting capable nodes to improve the system performance is thus demanded. For the security purpose the partitioned file chunks are stored in an encrypted format and also the main server can view the details of who download which file at which time. IndexTerms—Distributed File System (DFS),Map Reduce programming paradigm, Cloud. I