Harnessing the power of supercomputers using the PanDA Pilot 2 in the ATLAS Experiment

The unprecedented computing resource needs of the ATLAS experiment have motivated the Collaboration to become a leader in exploiting High Performance Computers (HPCs). To meet the requirements of HPCs, the PanDA system has been equipped with two new components; Pilot 2 and Harvester, that were designed with HPCs in mind. While Harvester is a resource-facing service which provides resource provisioning and workload shaping, Pilot 2 is responsible for payload execution on the resource. The presentation will focus on Pilot 2, which is a complete rewrite of the original PanDA Pilot used by ATLAS and other experiments for well over a decade. Pilot 2 has a flexible and adaptive design that allows for plugins to be defined with streamlined workflows. In particular, it has plugins for specific hardware infrastructures (HPC/GPU clusters) as well as for dedicated workflows defined by the needs of an experiment. Examples of dedicated HPC workflows in which the Pilot either acts like an MPI application that runs a set of jobs in an assembly, or by using the Yoda-Droid tools in the ATLAS Event Service mode, under the control of the Harvester service, are discussed. In addition to describing the technical details of these workflows, results are shown from its deployment on Cori (NERSC), Theta (ALCF), Titan and Summit (OLCF)