CFDComm: an optimized library for scalable point-to-point communication for general CFD applications

Domain decomposition is the most widely used technique to achieve parallelism in CFD applications. For complicated geometries usually graph partitioning programs are used to decompose the domain into smaller computational blocks such that the computation load is balanced and communication cost is minimized. In this paper an algorithm is provided and tested which avoids deadlocks in complicated communications patterns inherited from the graph decomposition process. The basic algorithm is implemented using FORTRAN 95 and MPI and then several optimization techniques are used to increase the scalability of the library which include addition of topologies, overlap of communication and computation to mask the message passing latency and non-blocking communication. The library is tested for up to 512 cores on the Iridis-3 cluster which incorporates 1008 compute nodes each composed of 2, 2.4 GHz 6-core Westmere processors. IO and inter-node communication is via a fast Infiniband network which is composed of groups of 32 nodes connected by DDR links to a 48 port QDR leaf-switch. The leaf switches then have 4 trunked QDR connections to 4 QDR 48-port core switches.