A Framework for Productive, Efficient and Portable Parallel Computing

Developing efficient parallel implementations and fully utilizing the available resources of parallel platforms is now required for software applications to scale to new generations of processors. Yet, parallel programming remains challeng- ing to programmers due to the requisite low-level knowledge of the underlying hardware and parallel computing constructs. Developing applications that effec- tively utilize parallel hardware is restricted by poor programmer productivity, low-level implementation requirements, and limited portability of the application code. These restrictions in turn impede experimentation with various algorithmic approaches for a given application. Currently, the programming world is divided into two types of programmers: application writers who focus on designing and prototyping applications and algorithms, and efficiency programmers who focus on extracting performance for particular compute kernels. The gap between these two types of programmers is referred to as "the implementation gap".In this dissertation, we present a software environment that aims to bridge the implementation gap and enable application writers to productively utilize parallel hardware by reusing the work of efficiency programmers. Specifically, we present PyCASP, a Python-based software framework that automatically maps Python application code to a variety of parallel platforms. PyCASP is an application- domain-specific framework that uses a systematic, pattern-oriented approach to offer a single productive software development environment for application writ- ers. PyCASP targets audio content analysis applications, but our methodology is designed to be applicable to any application domain. Using PyCASP, appli- cations can be prototyped in Python code and our environment enables them to automatically scale their performance to modern parallel processors such as GPUs, multicore CPUs and compute clusters. We use the Selective Embedded JIT Specialization (SEJITS) mechanism to realize the pattern-based design of PyCASP in software. We use SEJITS to implement PyCASP's components and to enable automatic parallelization of specific audio content analysis application patterns on a variety of parallel hardware. By focusing on one application domain, we enable efficient composition of computations using three structural patterns: MapReduce, Iterator and Pipe-and-Filter.To illustrate our approach, we study a set of four example audio content anal- ysis applications that are architected and implemented using PyCASP: a speaker verification system, a speaker diarization system, a music recommendation sys- tem and a video event detection system. We describe the detailed implementa- tion of two computational components of PyCASP: a Gaussian Mixture Model (GMM) component and a Support Vector Machine (SVM) component and their use in implementing the example applications. We also analyze composition of computations using the three structural patterns and implement the available optimizations for composing computations in audio analysis applications.We evaluate our approach with results on productivity and performance using the two computational components and the four example applications. Our re- sults illustrate that we can prototype the full-functioning applications in Python using 10 − 60× less lines of code than equivalent implementations using low-level languages. Our PyCASP components and example applications achieve and of- ten exceed the efficiency of comparable hand-tuned low-level implementations. In addition to specialization, adding the optimizations for composing components in these applications can give up to 30% performance improvement. We show that applications written using PyCASP can be run on multiple parallel hard- ware backends with little or no application code change. PyCASP also enables applications to scale from one desktop GPU to a cluster of GPUs with little pro- grammer effort. Combining all of the specialization and composition techniques, our example applications are able to automatically achieve 50-1000× faster-than- real-time performance on both multi-core CPU and GPU platforms and 15.5× speedup on 16-node cluster of GPUs showing near-optimal scaling