Leveraging Symmetric Multiprocessors and System Area Networks in Software Distributed Shared Memory

Clusters of workstations have long provided a cost-effective, large-scale parallel computing platform. A Software Distributed Shared Memory (SDSM) system simplifies programming on these platforms by presenting the illusion of shared memory. SDSM performance has historically been limited by the high cost of inter-processor communication overhead. Recent hardware trends, such as commodity symmetric multiprocessors (SMPs) and system area networks, can be used to potentially lower this overhead. The Cashmere SDSM has been designed for clusters of SMPs connected by a low-latency, remote-memory-access system area network. Cashmere introduces several novel techniques to leverage SMP hardware coherence and also to exploit remote-memory-access and other special features found in today's emerging system area networks. The results of our prototype implementation show that the Cashmere design leads to an average improvement of 25% over a comparable protocol version that does not leverage the SMP hardware coherence. The results also isolate the performance impact of various network features, thereby providing network designers with an informative application study. In addition, we have investigated the impact of these new hardware trends on the most fundamental aspect of SDSM design: the coherence granularity. Our findings show that recent hardware trends help reduce the performance gap between fine and coarse granularity SDSM. We also provide additional techniques for further reducing the gap