Vm-based shared memory on low-latency, remote-memory-access networks

Recent technological advances have produced network interfaces that provide users with very low-latency ac-cess to the memory of remote machines. We examine the impact of such networks on the implementation and performance of software DSM. Specifically, we compare two DSM systems—Cashmere and TreadMarks—on a 32-processor DEC Alpha cluster connected by a Memory Channel network. Both Cashmere and TreadMarks use virtual memory to maintain coherence on pages, and both use lazy, multi-writer release consistency. The systems differ dramati-cally, however, in the mechanisms used to track sharing information and to collect and merge concurrent updates to a page, with the result that Cashmere communicates much more frequently, and at a much finer grain. Our principal conclusion is that low-latency networks make DSM based on fine-grain communication compet-itive with more coarse-grain approaches, but that fur-ther hardware improvements will be needed before such systems can provide consistently superior performance. In our experiments, Cashmere scales slightly better than TreadMarks for applications with fine-grain interleaving of accesses to shared data. At the same time, it is severely constrained by limitations of the current Memory Channel hardware. On average, performance is better for Tread-Marks. Because most of its messages require remote pro-cessing, TreadMarks depends critically on the latency of interrupts: performance improves greatly when proces-sors poll for messages