Benchmarking Neural Networks on Heterogeneous Hardware Resources

In recent years, artificial intelligence (AI) became a key enabling technology for many domains. To achieve best performance, modern AI methods have high resource demands, e.g., GPU servers for the training of neural networks. With the advent of further processor technologies, such as tensor processors or re-wirable processors, AI methods may be executed in shorter time while even saving energy. For many application domains such as autonomous driving or unmanned aerial vehicles, real-time constraints mandate low end-to-end latencies in AI processing. In this paper, we present a combined micro- and macro-benchmarking approach to analyze the performance as well as the power demands of modern processor architectures using convolutional neural networks as workload. We discuss tradeoffs among the different processor types and indicate issues and challenges that arise when performing such benchmarks on heterogeneous hardware resources. We show that FPGAs allow for an increase of 7x up to 45x in performance over high-end GPUs while using only 10% of the power. In the consumer space, novel architectures such as the Apple M1 are able to offer 3-5x better performance at 10-20% the power draw of current x86 CPU or GPU hardware. This artifact contains the replication package for the respective paper (paper, slides included) on the Symposium of Software Performance 2021