Metrics and Benchmarking for Parallel Job Scheduling

Abstract. The evaluation of parallel job schedulers hinges on two things: the use of appropriate metrics, and the use of appropriate workloads on which the scheduler can operate. We argue that the focus should be on on-line open systems, and propose that a standard workload should be used as a benchmark for schedulers. This benchmark will specify dis-tributions of parallelism and runtime, as found by analyzing accounting traces, and also internal structures that create dierent speedup and syn-chronization characteristics. As for metrics, we present some problems with slowdown and bounded slowdown that have been proposed recently.