Hybrid performance-based workload management for multiclusters and grids

The paper addresses the dynamic scheduling of parallel jobs with quality-of-service demands (soft-deadlines) in multiclusters and grids. Three performance metrics (over-deadline, makespan and idle-time) are combined with variable weights to evaluate the scheduling performance. These three metrics are used to measure the extent to which jobs comply with their QoS demands, the resource throughput and the resource utilisation. Therefore, clusters situated in different administrative organisations can utilise different weight combinations to represent their different performance requirements. Two levels of performance optimisation are applied in the multicluster. At the multicluster level, a scheduler (which we call MUSCLE) allocates parallel jobs with high packing potential to the same cluster; MUSCLE also takes the jobs' QoS requirements into account and employs a heuristic to allocate suitable workloads to each cluster to balance the overall system performance. At the local cluster level, an existing workload manager, called TITAN, utilises a genetic algorithm to further improve the scheduling performance of the jobs sent by MUSCLE. Extensive experimental studies are conducted to verify the effectiveness of the scheduling mechanism in MUSCLE. The results show that, compared with traditional distributed workload allocation policies, the comprehensive scheduling performance (in terms of over-deadline, makespan and idle-time) of parallel jobs is significantly improved across the multicluster