Enabling and scaling matrix computations on heterogeneous multi-core and multi-gpu systems

We present a new approach to utilizing all CPU cores and all GPUs on heterogeneous multicore and multi-GPU systems to support dense matrix computations efficiently. The main idea is that we treat a heterogeneous system as a distributed-memory machine, and use a heterogeneous multi-level block cyclic distribution method to allocate data to the host and multiple GPUs to minimize communication. We design het-erogeneous algorithms with hybrid tiles to accommodate the processor heterogeneity, and introduce an auto-tuning method to determine the hybrid tile sizes to attain both high performance and load balancing. We have also imple-mented a new runtime system and applied it to the Cholesky and QR factorizations. Our approach is designed for achiev-ing four objectives: a high degree of parallelism, minimized synchronization, minimized communication, and load bal-ancing. Our experiments on a compute node (with two Intel Westmere hexa-core CPUs and three Nvidia Fermi GPUs), as well as on up to 100 compute nodes on the Keeneland sys-tem [31], demonstrate great scalability, good load balancing, and efficiency of our approach