Software-Controlled Transparent Management of Heterogeneous Memory Resources in Virtualized Systems

This paper presents a software-controlled technique for man-aging the heterogeneous memory resources of next gener-ation multicore platforms with fast 3D die-stacked mem-ory and additional slow off-chip memory. Implemented for virtualized server systems, the technique detects the ‘hot’ pages critical to program performance in order to then main-tain them in the scarce fast 3D memory resources. Chal-lenges overcome for the technique’s implementation include the need to minimize its runtime overheads, the lack of hypervisor-level direct visibility into the memory access be-havior of guest virtual machines, and the need to make page migration transparent to guests. This paper presents hypervisor-level mechanisms that (i) build a page access history of virtual machines, by periodically scanning page-table access bits and (ii) intercept guest page table operations to create mirrored page-tables and enable guest-transparent page migration. The methods are implemented in the Xen hypervisor and evaluated on a larger scale multicore plat-form. The resulting ability to characterize the memory be-havior of representative server workloads demonstrates the feasibility of software-managed heterogeneous memory re-sources