Software Shared Memory Support on Clusters of Symmetric MultiProcessors Using Remote-Write Networks

Low-latency, remote-write-access networks have recently become commodity items. These networks can connect clusters of symmetric multiprocessors (SMPs) to form very cost-effective, large scale parallel systems. Software-based distributed shared memory (SDSM) is a natural choice for the underlying platform. However, to exploit the platform’s full potential, sharing across SMPs must be managed without compromising the efficiency of sharing within an SMP. Cashmere-2L is a “two-level” SDSM protocol that delivers the platform’s potential through novel software techniques that leverage, without compromising, the efficiency of the hardware coherence. The protocol implements a moderately lazy release consistency model with page directories, home-nodes, and multiple concurrent writers. By avoiding global meta-data locks and TLB shootdown, Cashmere-2L is able to maintain a high level of asynchrony. The prototype Cashmere-2L system currently runs on an 8-node, 32-processor DEC AlphaServer cluster connected by a DEC Memory Channel remote-memory-acccess network. Across nine applications, the system obtains speedups from 5 to 26 on 32 processors. We quantify the performance of protocol optimizations and alternatives along two dimensions – SMP exploitation and remote-write network utilization. To determine the effectiveness of our two-level design, we compare its performance with a one-level protocol that does not leverage hardware coherence, and evaluate the benefits of our protocol’s asynchrony. We also compare three alternative protocols that utilize the remote-write network to varying degrees: placin