A BF16 FMA is all you need for DNN training

Fused Multiply-Add (FMA) functional units constitute a fundamental hardware component to train Deep Neural Networks (DNNs). Its silicon area grows quadratically with the mantissa bit count of the computer number format, which has motivated the adoption of the BrainFloat16 format (BF16). BF16 features 1 sign, 8 exponent and 7 explicit mantissa bits. Some approaches to train DNNs achieve significant performance benefits by using the BF16 format. However, these approaches must combine BF16 with the standard IEEE 754 Floating-Point 32-bit (FP32) format to achieve state-of-the-art training accuracy, which limits the impact of adopting BF16. This article proposes the first approach able to train complex DNNs entirely using the BF16 format. We propose a new class of FMA operators, FMAbf16 n m, that entirely rely on BF16 FMA hardware instructions and deliver the same accuracy as FP32. FMAbf16 n m operators achieve performance improvements within the 1.28- 1.35X range on ResNet101 with respect to FP32. FMAbf16 n m enables training complex DNNs on simple low-end hardware devices without requiring expensive FP32 FMA functional units.Marc Casas was partially supported by the under Grant RYC-2017-23269 funded by MCIN/AEI/10.13039/501100011033, and by ESF Investing in your future. Adrià Armejach is a Serra Hunter Fellow and has been partially supported by the under Grant IJCI-2017-33945 funded by MCIN/AEI/10.13039/501100011033. John Osorio has been partially supported by the under Grant PRE2019-090406 funded by MCIN/AEI/10.13039/501100011033 and by ESF Investing in your future. This work has been partially supported by Intel under the BSC-Intel collaboration and European Union Horizon 2020 research and innovation programme under Grant 955606 - DEEP-SEA EU project.Peer ReviewedPostprint (author's final draft