JETS: Language and system support for many-parallel-task workflows

Many-task computing is a well-established paradigm for implementing loosely coupled applications on large-scale computing systems. However, few of the model’s exist-ing implementations provide efficient, low-latency support for the execution of tightly coupled applications as atomic tasks. Thus, a vast array of parallel applications cannot readily be used effectively within many-task workloads. In this work, we present JETS, a middleware component that provides high performance support for many-parallel-task-computing (MPTC). JETS is based on a highly concurrent approach to parallel task dispatch and on new capabili-ties now available in the MPICH2 MPI implementation and the ZeptoOS Linux operating system. JETS represents an advancement over the few known examples of multi-level many-parallel-task scheduling systems by more efficiently scheduling many short-duration parallel application invo-cations; by overcoming the challenges of coupling the user processes of each application invocation via the messaging fabric; and by concurrently managing many application ex-ecutions in various stages. We report here on the JETS architecture and its performance on both synthetic bench-marks and the NAMD molecular dynamics application.