1

Runtime systems are critical in the support for big data applications. For example, task dependency management needs to be done in a distributed fashion (in order to offer scalable support for data-flow driven programming systems with massive dependency graphs). Scheduling needs to be done in a distributed fashion, as single gateway nodes will not be able to manage millions of jobs to be scheduled to billions of cores; it would be both computationally prohibitive (most scheduling algorithms are NP-complete), and communication prohibitive (maintaining the state of millions of active state-full network connections is not feasible with today’s systems). In order to hide latency, we need to overlap computations with communication. In order to better support dynamic systems, run-time systems should also support dynamic provisioning, allowing the system to be easier to make resilient as well as making it usable in a cloud computing environment. We need the framework to support heterogeneous hardware (e.g. accelerators with many-core architectures) to ensure it is compatible with the growing segment of high-end computing. And run-time systems need HPC support, in order to have a broader appeal to traditional high-end computing systems and applications. This work will extend two existing research projects, namely Swift [2, 3, 5] and MATRIX [9] with the following enhancements. Distributed Scheduling: In order to improve both performance and reliability, clients will be able t