Distributed computing and communication in peer-to-peer networks

Traditional distributed computing systems are typically complex to implement and costly to maintain. Furthermore, little comparative work has been done to understand the performance and usability of these systems for their own sake as well of that of new approaches that eventuate. The work presented herein addresses both of these problems by describing the design and implementation of CompTorrent, a simple to implement and maintain decentralised peer-to-peer computing network, based on techniques used in other, non-computing peer-to-peer systems. This research also describes a new framework (WAN-DC) suitable for the comparison of wide area distributed computing systems. CompTorrent is compared with BOINC and Condor, two traditional distributed computing systems, using the WAN-DC framework on the same cluster hardware. WAN-DC consists of a baseline to quantify the size of the cluster hardware followed by a set of well-known algorithms including calculations of the Mandelbrot set, a conversion of video formats (Transcode) and a ray-tracing of a benchmarking scene (POV- Ray). Other experiments include determining the systems underlying overhead with work units of no load (No Work), as well as work units of ranging sizes in order to measure where a system becomes acceptably efficient. This last test, One Second, is particularly useful when comparing different systems and approaches. Results show that CompTorrent maintains a performance range between that of BOINC and Condor for all cluster node sizes for the POV-Ray experiment. Transcode shows CompTorrent is between or better than BOINC and Condor in 50% of cases and, whilst worse in the other half of experiments, it was only by approximately 15% in the worst case. Mandelbrot showed results between both distributed systems and, similarly to POV-Ray, No Work behaves either between or better than the results of both BOINC and Condor. These, and other, results described within, show that that a distributed processing system based on a decentralised, peer-to-peer network can provide similar results to distributed processing systems based on traditional client/server networking architectures. This work demonstrates a convergence of peer-to-peer and distributed computing systems which while considered as certain as "death and taxes" (Foster & Iamnitchi 2003) has not, until now, been formally demonstrated in an academic setting for general purpose distributed computing where comparable systems exist based on a client-server approach. It is hoped that this work contributes to the further adoption of "Grassroots" distributed computing by bringing the ability to host and manage a distributed computing project to a much wider audience