PARRAY: A unifying array representation for heterogeneous parallelism

This paper introduces a programming interface called PARRAY (or Parallelizing ARRAYs) that supports system-level succinct pro- griming for heterogeneous parallel systems like GPU clusters. The current practice of software development requires combin- ing several low-level libraries like Pthread, OpenMP, CUDA and MPI. Achieving productivity and portability is hard with different numbers and models of GPUs. PARRAY extends mainstream C pro- gramming with novel array types of the following features: 1) the dimensions of an array type are nested in a tree structure, con- ceptually reflecting the memory hierarchy; 2) the definition of an array type may contain references to other array types, allowing so- phisticated array types to be created for parallelization; 3) threads also form arrays that allow programming in a Single-Program- Multiple-Codeblock (SPMC) style to unify various sophisticated communication patterns. This leads to shorter, more portable and maintainable parallel codes, while the programmer still has con- trol over performance-related features necessary for deep manual optimization. Although the source-to-source code generator only faithfully generates low-level library calls according to the type in- formation, higher-level programming and automatic performance optimization are still possible through building libraries of sub- programs on top of PARRAY. The case study on cluster FFT illustrates a simple 30-line code that 2x-outperforms Intel Cluster MKL on the Tianhe-1A system with 7168 Fermi GPUs and 14336 CPUs. Copyright ? 2012 ACM.EI