The Data Transfer Engine: Towards a Software Controlled Memory Hierarchy

In today’s computer architectures, many scientific applications are considered to be memory bound. The memory wall, i.e. the large disparity between a processor’s speed and the required time to access off-chip memory, is a yet-to-be-solved problem that can greatly reduce performance and make us underutilise the processors capabilities. Many different approaches have been proposed to tackle this problem, such as the addition of a large cache hierarchy, multithreading or specula-tive data prefetching. Most of these approaches rely on the prediction of the application’s future behaviour, something that should not be necessary as this information is known by the program-mer and is located in the application itself. Instead of designing hardware that attempts to guess the future, the goal should be to provide the programmer with the hardware support required to decide when the data is transferred and where is it transferred to. With this goal in mind, we introduce the Data Transfer Engine, a runtime-assisted, software prefetcher that exploits the in-formation provided by the programmer in order to place data in the cache hierarchy close to the processor that will make use of it. The DTE can not only significantly reduce stall time due to cache misses but, more importantly, it allows us to design new computer architectures that are able to tolerate very high memory latencies.