Agile parallel applications

Non-dedicated loosely coupled systems are popular platforms for cluster- and grid-based parallel processing, fundamentally because they have good cost–performance ratios and are scalable. However, these platforms represent highly dynamic environments in which performance and efficiency can be seriously impacted by changes in environmental conditions. This is especially significant where the runtime configuration has been determined statically, either at compilation time or at the start of execution. This paper introduces the concept of agile parallel processing in which the application manages several aspects of its own run-time behaviour, including deployment granularity. This approach reduces the emphasis on the preconfiguration of components, and relies instead on inbuilt learning and discovery capabilities. To facilitate investigation into the extent to which a self-managing approach can be beneficial to parallel processing, an experimental framework has been developed. The framework provides a range of services such as dynamic worker discovery and performance calibration, and policy controlled facilities such as resource management and adaptation to suit environmental conditions. The framework integrates these services with the parallel application code. The operation and performance of policy-based dynamic deployment scheduling in dynamic environments is analysed in detail