Ingebed Deep Learning - Algoritmes, architecturen en circuits voor het continu verwerken van neurale netwerken

Although state-of-the-art in many typical machine-learning tasks, deep learning algorithms are very costly in terms of energy consumption, due to their large amount of required computations and huge model sizes. Because of this, deep learning applications on battery constrained wearables have only been possible through wireless connections with a resourceful cloud. This setup has several drawbacks. First, there are privacy concerns. Cloud computing requires users to share their raw data - images, video, locations, speech - with a remote system. Most users are not willing to do this. Second, the cloud-setup requires users to be connected all the time, which is unfeasible given current cellular coverage. Furthermore, real-time applications require low latency connections, which cannot be guaranteed using the current communication infrastructure. Finally, wireless connections are very inefficient - requiring too much energy per transferred bit for real-time data transfer on energy constrained platforms. All these issues - privacy, latency/connectivity and costly wireless connections - can be resolved by moving towards computing at the edge. Finding ways to do this is the main topic of this dissertation. It focuses on techniques to minimize the energy consumption of deep learning algorithms for embedded applications on battery-constrained wearable edge devices. Computing in the edge is only possible if these deep learning algorithms can be run in a more energy-efficient way, within the energy- and power-budget of the computing platform available on a wearable device. In order to achieve this, several innovations are necessary on all levels of an application's design hierarchy. Smarter applications can be developed for more statistically efficient deep learning algorithms, which in turn should run on optimized hardware platforms built on specifically tailored circuits. Finally, designers should not focus on any of these fields separately, but should co-optimize hardware and software to create minimum energy deep learning systems. I've tried contributing to all of these fields in this dissertation. On the application level this work proposes hierarchical cascaded processing, a generalization of wake-up based systems into multi-stage cascades. As the technique allows better exploitation of input data statistics of sparse data, such as video and speech, it allows reducing the energy consumed in a full face-recognition application by up to 2 orders of magnitude compared to a wake-up based baseline. This is an example of how deep learning can be brought to edge devices by creating smarter applications. On the algorithmic level this dissertation focuses on minimizing energy consumption of neural networks by exploiting their fault-tolerance, through precision scaling, and sparsity, through computation skipping. Depending on the application, networks require varying computational precision, leading to different levels of sparsity. This dissertation first shows how hardware platforms can be adapted to benefit from this, showing up to 30x gains compared to a full precision baseline. On top of that network architectures and the hardware they would run on can be co-designed by training Quantized Neural Networks, networks trained for arbitrarily low precision from scratch. Using this approach, the network architecture, the on-chip memory, the used number of bits and the level of on-chip parallelism can simultaneously be chosen to minimize energy consumption given a specific accuracy on a benchmark. The approach shows 3-10ximprovements by going to optimized 4b networks over 8b and 16b implementations. On the circuit level, the text discusses Dynamic-Voltage-Accuracy-Frequency-Scaling (DVAFS), the most efficient method to date to dynamically trade-off computational accuracy for energy consumption. As such, it is a contribution to the field of approximate computing. As became apparent from studies on the algorithmic level, neural network hardware platforms can benefit from the DVAFS technique to minimize energy consumption of different neural network with different computational precision requirements. This dissertation focuses on the performance of DVAFS in comparison with other approximate computing techniques (showing 3-5x gains at iso-accuracy) and discusses its implementation challenges. Based on the observations made on the application, algorithmic and circuit level, several silicon prototypes have been designed. The Envision processors are built to minimize energy consumption in sparse, low-precision CNNs. (a) They employ a novel 2D-SIMD MAC-array to maximize data reuse for convolutional neural network workflows. (b) They exploit ReLU and quantization generated sparsity by guarding unnecessary memory reads and preventing shutting down rows and columns in the 2D MAC-array for sparse input data. (c) They modulate their energy consumption with varying computational precision to minimize energy for a network with any requirements. DVAS-compatible Envision V1 was implemented in a 40nm technology on 2.4mm2. Through DVAS and sparsity guarding, it can bring down the processors power consumption down from 278mW to 55mW (5x) at the same throughput, while losing just 1% of relative benchmark accuracy. At the time of publication, Envision V1 was 5x more efficient than other published alternatives. DVAFS-compatible Envision V2 was implemented in a 28nm FD-SOI technology on 2mm2. As DVAFS allows for larger system-level gains, it can bring down power consumption from 290mW to 20mW (14.5x) at the same throughput and 1% accuracy loss. At the time of publication, Envision V2's ultimate mode was 20% more efficient than its closest competitor. The Envision processors operate in the 10-300mW power range for mobile applications. They are meant to be used in the later stages of a cascaded application. The BinarEye processors are fully optimized for BinaryNets, networks with all weights and activations constrained to +/-1. These chips target a wide variety of always-on visual wake-up applications on battery-constrained mobile devices, typically the earliest stages in the hierarchical processing cascades. This goal of always-on compute is achieved through: (1) extremely efficient hardware in the analog domain, (2) maximal reuse of weights in a memory-like neuron array (3), storing full models and feature maps on chip, requiring no off-chip bandwidth and (4) flexibility on three levels: retrainable weights, a programmable network depth and a reconfigurable network width. Two architectures exploiting at least three of the previous characteristics are presented and implemented. MSBNN, a Mixed-Signal version, consumes only 3.8uJ per input-to-label classification, while achieving 237fps and 86% accuracy on the CIFAR-10 benchmark. BinarEye, a fully digital version, consumes 14.5uJ for the same specifications, but can scale down to 0.92uJ per classification at 76% and 3800 fps. Both versions hereby outperform the state-of-the-art 3-40x at similar accuracy. Apart from a case-study on analog in-memory compute in neural networks, this work is also a first direct measured comparison of the efficiency of digital versus analog computations.status: accepte