A Dynamic, Multi-Protocol Data Storage Integration Framework for Multi-Tenanted Science Gateway Middleware

Science gateways are user-centered, end-to-end cyberinfrastructure for managing scientific data and executions of computational software on distributed high performance computing and cloud resources. A typical goal of a science gateway is to help users make more effective use of high performance and other research computing resources, thus easing the user support burden on research computing centers while also increasing their ability to scalably serve user communities. An important challenge for science gateways is to manage data at scale, where scaling pressures come from both the number of users of successful gateways and the size of data used in scientific workflows that the gateways execute. This paper examines the use of managed file transfer (MFT) approaches to generalize several data flow scenarios. We examine a particular implementation, Airavata MFT, which can be used to extend the data stores integrated with a science gateway beyond local storage to include multiple remote storage instances, including cloud vendors. An agent-based approach is adopted, which enables streamlined end-to-end flows of data between user storage and backend computing. Integration with these diverse storage types is done through a common application programming interface for data operations, making the integration of a new storage system a configuration change without requiring changes to portal server code