Extending CloudKon to Support HPC Job Scheduling

Abstract — With the ongoing trends in computing and data systems, the time is not far when we will have exascale systems with millions of nodes and threads of execution being the major driving force for most of the large scale, distributed operations. Today, job management systems need to support a variety of applications, such as Many-Task Computing- MTC and; High-Performance Computing- HPC, that have a bulk of tasks with finer granularity due to such high parallelism supported. Most of the popular systems today, have Master/Slave architectures where a centralized server is responsible for all resource management and job executions, making it a single point of failure and also inefficient to scale at petascale systems that require finer granular workloads. The goal of this project is to enhance the recently developed CloudKon system to support HPC jobs using the various public cloud services (Amazon's SQS, DynamoDB and EC2), and distinguishing HPC and MTC tasks by the number of cores/nodes a task requires. This would work in cohesion with the actual motive behind developing CloudKon, which is to act as a distributed job management system that can support millions of tasks from multiple users delivering over 2X the performance compared to other state-of-the-art systems in terms of throughput