The effect of contention on the scalability of page-based software shared memory systems

We demonstrate the profound effects of contention on the performance of page-based software distributed shared memory systems, as such systems are scaled to a larger number of nodes. Programs whose performance scales will experience only minor increases in memory latency, do not suffer from contention, and show a balanced communication load. In contrast, programs that scaled poorly suffered from large memory latency increases due to contention and communication imbalance. We use two existing protocols, Princeton's home-based protocol and the TreadMarks protocol, and a third novel protocol, Adaptive Striping. For most of our programs, all three protocols were equally affected by latency increases and achieved similar performance. Where they differ significantly, the communication load imbalance, which is caused by the read accesses to pages that have multiple readers following one or more writers, is the largest factor accounting for the difference