Using Containers in HPC for Research Workflows

At Pawsey Supercomputing Centre we support researchers with the maintenance of a wide variety of algorithms suitable for execution on our various systems. As the diversity of research domains that require access to high performance computing increases we have found that we are encountering an increasing number of applications that require our maintenance and oversight. We need software management solutions that are scalable, supportable and useful to users. We have deployed container solutions across our systems to partially address these issues while dealing with a rapidly evolving software stack. We will discuss our container solutions at Pawsey and describe how we have deployed these in collaboration with users across Cloud and HPC systems. These solutions can provide improved user control over software deployment, improved research reproducability and can improve algorithm performance for suitable research use cases. ABOUT THE AUTHOR(S) Mark Gray is the Cloud Lead at Pawsey Supercomputing Centre where he manages Nimbus, a national cloud service for researchers in Australia. Mark has experience in DevOps roles at Pawsey, IMOS, AusCover and NASA Goddard Space Flight Center. Mark represents Pawsey Supercomputing Centre both nationally and internationally, with respect to the use of Nimbus. He also provides project management skills for projects of key strategic importance to the Centre. Mr Gray brings a strong research and data management background to his role of managing all aspects of this service. This includes procurement, deployment, training, user expectation management – and leadership of the team who administer and operate the service. Marco De La Pierre completed a PhD in Materials Science, specialising in theoretical and computational chemistry. Joining Pawsey in 2018, Marco engages with researchers in the fields of computational materials science, computational chemistry and bioinformatics. Marco’s area of expertise is quantum mechanical calculations and classical molecular dynamics (including free energy calculations). He is also highly experienced in software development (mostly Fortran, plus a bit of C++) and has accumulated an extensive skill set of bash/Python scripting to automate workflows for pre-processing, post-processing and visualisation of simulation data. Marco was also employed for a number of years as a university researcher, developing and applying methods to model, simulate and analyse materials properties and processes. Brian Skjerven is a Supercomputing specialist with a Master of Science in Applied Mathematics and a Bachelor of Administration Mathematics. In 2014 he was a visiting researcher for CSIRO and in the same year was employed by Pawsey in his current position of Supercomputing Application Specialist, where he assists regional and national users with Pawsey HPC resources. He is also responsible for conducting user training sessions for Pawsey Supercomputers, he acts as lead for developing and implementing container solutions for users of Pawsey Cloud and HPC systems and mentors and trains undergraduate students selected by Pawsey’s Summer Internship program and Student Cluster Competition team.