Efficient On-Demand Operations in Large-Scale Infrastructures

In large-scale distributed infrastructures such as clouds, Grids, peer-to-peer systems, and wide-area testbeds, users and administrators typically desire to perform on-demand operations that deal with the most up-to-date state of the infrastructure. However, the scale and dynamism present in the operating environment make it challenging to support on-demand operations efficiently, i.e., in a bandwidth- and response-efficient manner. This dissertation discusses several on-demand operations, challenges associated with them, and system designs that meet these challenges. Specifically, we design and implement techniques for 1) on-demand group monitoring that allows users and administrators of an infrastructure to query and aggregate the up-to-date state of the machines (e.g., CPU utilization) in one or multiple groups, 2) on-demand storage for intermediate data generated by dataflow programming paradigms running in clouds, 3) on-demand Grid scheduling that makes worker-centric scheduling decisions based on the current availability of compute nodes, and 4) on-demand key/value pair lookup that is overlay-independent and perturbation-resistant. We evaluate these on-demand operations using large-scale simulations with traces gathered from real systems, as well as via deployments over real testbeds such as Emulab and PlanetLab