Experiences with self-organizing, decentralized grids using the grid appliance

“Give a man a fish, feed him for a day. Teach a man to fish, feed him for a lifetime ” – Lau Tzu Large-scale grid computing projects such as TeraGrid and Open Science Grid provide researchers vast amounts of com-pute resources but with requirements that could limit ac-cess, results delayed due to potentially long job queues, and environments and policies that might affect a user’s work flow. In many scenarios and in particular with the advent of Infrastructure-as-a-Service (IaaS) cloud computing, indi-vidual users and communities can benefit from less restric-tive, dynamic systems that include a combination of local resources and on-demand resources provisioned by one or more IaaS provider. These types of scenarios benefit from flexibility in deploying resources, remote access, and envi-ronment configuration. In this paper, we address how small groups can dynami-cally create, join, and manage grid infrastructures with low administrative overhead. Our work distinguishes itself from other projects with similar objects by enabling a combina-tion of decentralized system organization and user access for job submission in addition to a web 2.0 interfaces for manag-ing grid membership and automate certificate management. These components contribute to the design of the “Grid Ap-pliance, ” an implementation of a wide area overlay network of virtual workstations (WOW), which has developed over the past six years into a mature system with several de-ployments and many users. In addition to an architectural description, this paper contains lessons learned during the development and deployment of “Grid Appliance ” systems and a case study backed by quantitative analysis that veri-fies the utility of our approach