Efficient Communication Between Parallel Programs with InterComm

We present the design and implementation of InterComm, a framework to couple parallel components that enables efficient communication in the presence of complex data distributions within a coupled application. Multiple parallel libraries and languages may be used in different modules of a single application. The ability to couple such modules is required in many emerging application areas, such as complex simulations that model physical phenomena at multiple scales and resolutions, and remote sensing image data analysis applications. The complexity of the communication algorithms is highly dependent on the distribution of data across the processes in a distributed memory parallel program. We classify the distributions into two types - one that represents a data distribution in a compact way so that the distribution information can be replicated, and one that explicitly describes the location of each data element, so can be very large, requiring that the distribution information be distributed across processes as is the data. InterComm builds on our previous work on the Meta-Chaos program coupling framework. In that work, we showed that the steps required to perform the data exchanges include locating the data to be transferred within the local memories of each program, generating communication schedules (the patterns of interprocessor communication) for all processes, and transferring the data using the schedules. In this paper we describe the new algorithms we have developed, and show how those algorithms greatly improve on the Meta-Chaos algorithms by reducing the high cost of building the communication schedules. We present experimental results showing the performance of various algorithmic tradeoffs, and also compare against the original Meta-Chaos implementation. (UMIACS-TR-2004-04