Asynchronous Communication for Finite-Difference Simulations on GPU Clusters using CUDA and MPI

Graphical processing Units (GPUs) are finding widespread use as accelerators in computer clusters. It is not yet trivial to program applications that use multiple GPU-enabled cluster nodes efficiently. A key aspect of this is managing effective communication between GPU memory on separate devices on separate nodes. We develop a algorithmic framework for Finite-Difference numerical simulations that would normally require highly synchronous data-parallelism so they can effectively use loosely coupled GPU-enabled clus-ter nodes. We employ asynchronous communications and appropriate memory overlay of computations and communications to hide latency