Grenoble INP

It is well known that the placement of threads and memory plays a crucial role for performance on NUMA (Non-Uniform Memory-Access) systems. The conven-tional wisdom is to place threads close to their memory, to collocate on the same node threads that share data, and to segregate on different nodes threads that com-pete for memory bandwidth or cache resources. While many studies addressed thread and data placement, none of them considered a crucial property of modern NUMA systems that is likely to prevail in the future: asymmetric interconnect. When the nodes are connected by links of different bandwidth, we must consider not only whether the threads and data are placed on the same or different nodes, but how these nodes are connected. We study the effects of asymmetry on a widely avail-able x86 system and find that performance can vary by more than 2 × under the same distribution of thread and data across the nodes but different inter-node connectiv-ity. The key new insight is that the best-performing con-nectivity is the one with the greatest total bandwidth as opposed to the smallest number of hops. Based on our findings we designed and implemented a dynamic thread and memory placement algorithm in Linux that delivers similar or better performance than the best static place-ment and up to 218 % better performance than when the placement is chosen randomly.