Building a Sparse Convolutional Neural Network Inference Accelerator on Field-programmable Gate Arrays

High computational complexity and large memory footprint hinder the adoption of convolution neural networks (CNNs) in latency sensitive applications, such as self-driving. The presence of many zeros in CNN parameters and intermediate results -- a phenomenon known as sparsity -- presents opportunities for hardware accelerators to circumvent these problems, leading to speed-up. Prior work on accelerating CNNs using field-programmable gate arrays (FPGAs) has mostly omitted the benefits from sparsity. The few exceptions either leverage sparsity only on fully-connected layers in CNNs, or scale poorly due to complicated datapath and control logic. In this thesis, we propose a fine-grained structured weight sparsity for accelerating CNNs on FPGAs with relative ease. To leverage this sparsity for CNN acceleration on FPGAs, we design a novel processing element (PE) and an end-to-end sparse CNN inference accelerator that arranges the PEs in a systolic-array. Our results show that pruning according to the proposed structured weight sparsity pattern can reduce the amount of multiply-accumulate (MAC) operations in VGG-16 and ResNet-50 v.1.5 by 3.34 fold and 2.69 fold, respectively, while keeping the top-1 validation accuracy on ImageNet within 5\%. When implemented on an Intel Arria 10 GX1150 FPGA, our sparse CNN inference accelerator can achieve 2.26 TOP/s and 1.17 TOP/s on the MAC operations in the pruned VGG-16 and ResNet-50 v1.5, respectively. These figures translate to 2.86 fold and 1.69 fold speed-up over a baseline accelerator that is agnostic to sparsity.M.A.S