Scalable bioinformatics workflows for growing '-omics' datasets on National Collaborative Research Infrastructure Strategy Facilities

This presentation took place during a session at the eResearch Australasia 2021 Conference 11-15 October 2021. A recording of this presentation is available on the Australian BioCommons YouTube Channel. Abstract: Introduction Australian researchers rely on High Performance Computing (HPC) to process increasingly large life science and ‘-omics’ datasets. Bioinformatics workflows are often complex and, unlike traditional HPC, orchestrate multiple data, memory, I/O, and/or time-intensive compute tasks that mismatch local infrastructure paradigms. As a part of the Australian BioCommons, we have re-engineered popular bioinformatics workflows to enable scalable and efficient use of National Collaborative Research Infrastructure Strategy (NCRIS) eResearch facilities. Methods Re-engineered workflows and end-user resources were developed to meet FAIR data principles. Workflows identified through national surveys were optimised in collaboration with multiple research groups and national compute facility specialists. This ensured user-friendliness, applicability across research interests, datasets, and optimal use of local HPC architecture. Workflows are registered on WorkflowHub for findability and pipeline adoption is supported through training, user guides and documentation that follow FAIR recommendations by the Australian BioCommons. Results Developed workflows are available through the Australian BioCommons WorkflowHub space (https://workflowhub.eu/programmes/8). Between 2019-21, optimised workflows were successfully adopted by independent groups across pathogen, human, domestic animal, agricultural, and wildlife research. They have contributed to successful Australian competitive grants, national/local HPC allocation applications and research publications. Conclusions Creating efficient and scalable bioinformatics workflows on Australian compute infrastructure requires custom development to meld specific infrastructure hardware, usage and access policies with domain best practices. We are refining support and ongoing maintenance models and aim to enhance existing pipelines by adding auto-deployment capabilities and portability to other compute infrastructure accessible to Australian researchers