Deduplication in Distributed Storage Systems

As storage capacity requirements grow, storage systems are becoming distributed, and that distribution poses a challenge for space savings processes. In this thesis, I design and implement a mechanism for storing only a single instance of duplicated data within a distributed storage system which selectively performs deduplication across each of the independent computers, known as nodes, used for storage. This involves analyzing the contents of each node for objects with characteristics more likely to have duplicates elsewhere, particularly using duplication within a node as the indicative property- an object duplicated many times in a dataset will likely be duplicated at least once in some node. An inter-node system is responsible for efficiently collecting and distributing the information of these potential duplicates. A test implementation was built and run on several data sets characteristic of commo