Enabling dynamic interactions in large scale applications and scientific workflows using semantically specialized shared DataSpaces

Emerging scientific and engineering applications use large-scale parallel machines to simulate, with higher accuracy, complex physical phenomena consisting of dynamically interacting processes. The workflows associated with these applications consist of parallel application codes that need to co- ordinate and interact at runtime. The interactions typically involve large volumes of data that must be exchanged and processed by the codes. The heterogeneous nature of the coupled codes, their numerical formulations, and their data decompositions lead to complex and dynamic interaction and data exchange patterns that are only defined at runtime. Moreover, these simulations often run on separate resources and progress at different rates, which adds to their complexity. Efficient and scalable implementation of these coupled application workflows present several challenging programming, orchestration, coordination, and data exchange requirements. Existing programming frameworks, however, are rigid and provide limited support for the dynamic inter- actions manifested by these applications. For example, existing frameworks need to gather global application knowledge, impose tight synchronization between applications, or demand pre-defined and static interaction patterns that must be known prior to execution. These constraints can introduce significant performance penalties and can limit application interaction programming expressiveness. This thesis explores a new communication and coordination model to enable flexible and asynchronous application coupling for coupled applications workflows. It derives from the tuple-space model and provides the abstraction of a virtual distributed shared-space, which is customized for the application data domain. It enables applications to coordinate and exchange data by inserting and retrieving data objects. This model does not impose any synchronization requirements between independent applications. Data stored on the space can be accessed by multiple applications, which can associatively query the space and retrieve data objects. Furthermore, it enables decoupled and dynamic interactions driven by application computations. This thesis presents DataSpaces, a prototype implementation of the distributed shared-space model. DataSpaces enables memory-to-memory application coupling and transparent data redistribution. It can complement existing workflow engines to enable in-memory data transports between distributed applications that run on separate resources as part of end-to-end scientific workflows. The thesis also presents ActiveSpaces, which extends DataSpaces and the shared-space model to enable in-transit data processing. It proposes and demonstrates a shift in the data processing paradigm by moving processing code closer to the data. ActiveSpaces provides programming support for defining data processing routines, and a runtime execution system to deploy and remotely execute these routine on the space. The research concepts and software frameworks have been deployed and evaluated using real application workflows in production runs on high-end computing systems.Ph. D.Includes bibliographical referencesIncludes vitaby Ciprian Doca