One Step Closer to Converged Computing: Achieving Scalability with Cloud-Native HPC

Scripts, YAMLs, raw output data, and instructions for reproducing our CANOPIE 22 paper. The paper abstract is below. As High Performance Computing (HPC) workflows increase in complexity, their designers seek to enable automation and flexibility offered by cloud technologies. Container orchestration through Kubernetes enables highly desirable capabilities but does not satisfy the performance demands of HPC. Kubernetes tools that automate the lifecycle of Message Passing Interface (MPI)-based applications do not scale, and the Kubernetes scheduler does not provide crucial scheduling capabilities. In this work, we detail our efforts to port CORAL-2 benchmark codes to Kubernetes on IBM Cloud and AWS EKS. We describe contributions to the MPI Operator to achieve 3,000-rank scale, a two-orders-of-magnitude improvement to state of the art. We discuss enhancements to Fluence, our scheduler plugin for Kubernetes based on the next-generation, cloud-ready Flux framework. Finally, we compare the placement decisions of Fluence with those of the Kubernetes scheduler and demonstrate that Fluence allows simulated scientific workflows to achieve up to 3x higher performance.Corresponds to GitHub repo tag https://github.com/flux-framework/flux-k8s/releases/tag/v1.0.