On automation for optimised and robust deployment of neural networks on edge devices

Embedded systems are becoming interconnected and collaborative systems able to perform autonomous tasks. The remarkable expansion of the embedded and IoT market, together with the rise and breakthroughs of deep learning, has put the focus on the Edge as it stands as one of the keys for the next technological revolution: the seamless integration of artificial intelligence in our daily life. However, porting deep learning methods to edge devices poses several challenges due to their limited on-board storage and computing capabilities. The deployment of such methods - meeting latency and memory constraints - on a given target platform requires a complex optimisation process that becomes a bottleneck for end users. Moreover, deep learning methods deployed in embedded environments lack predictive robustness as the data distribution learned in off-line training might not match the deployment environment’s underlying distribution. This thesis addresses these challenges by automating the deployment of deep neural networks (DNNs) to obtain custom, robust, and efficient solutions on edge devices. We focus on connecting the various elements of an end-to-end development and automating the optimisation process to explore an ample design space while reducing the needed expertise. In addition, we extend the traditional deep learning flow by introducing feedback signals in the development process to address the lack of optimisation and robustness for custom deep learning applications on a given target environment. The thesis starts by evaluating deep learning frameworks – both high-performance computing and edge-oriented – for the training and deployment of DNNs. We verify that efficient development of deep learning solutions for custom applications on embedded devices requires a fine-grained integration of data, algorithms, and deployment tools. Thus, we present a modular AI pipeline as an integrating framework and investigate the several steps that lead to an end-to-end development. We put a particular focus on the deployment step where we present our lightweight deployment framework: LPDNN. We showcase the AI pipeline by providing several deployment examples of deep learning tasks, e.g., keyword spotting, image classification, and object detection, on a set of well-known embedded platforms, where LPDNN consistently outperforms all other popular deployment frameworks. In the second part of the thesis, we deepen into the deployment optimisation of DNNs across the different levels of software stack. We thoroughly review the literature and find a lack of research on cross-level optimisation as the space of approaches becomes too large to test and obtain a globally optimised solution in terms of latency, accuracy, and memory. Building on this knowledge, we present an automated exploration framework, including the target hardware-in-the-loop, to ease the deployment of DNNs. The framework automatically explores the design space on a embedded platform and finds an optimised solution that speeds up the performance and reduces the memory. Thus, we present a set of results for state-of-the-art DNNs on a heterogeneous (GPGPU) platform and a on range of Arm Cortex-A CPU platforms, achieving up to 4x improvement in performance and over 2x reduction in memory with negligible loss in accuracy. In the last part of the thesis, we revisit the end-to-end pipeline and introduce software optimisations and feedback signals to obtain a robust and efficient deployment on a low-power autonomous driving mini-vehicle. First, we verify the challenges to scale deep learning methods to micro-controller units (MCUs) and realise the lack of robustness when neural networks are exposed to embedded environments. Hence, we propose a closed-loop learning flow that includes the deployment environment-in-the-loop. We train a family of tiny neural networks (tinyCNN) to control the mini-vehicle, which gain robustness to lighting conditions over time. Further, we leverage GAP8, a parallel ultra-low-power RISC-V MCU, to meet the inference requirements, and introduce an online predictor that can choose between different tinyCNN models at runtime - trading accuracy and latency