Heterogeneous Model to Heterogeneous System Mapping with Computation and Communication Awareness

While machine learning (ML) has been widely used in real-life applications, the complex nature of real-world problems calls for heterogeneity in both machine learning models and hardware systems. For the algorithm, the heterogeneity in ML models comes from the multi-sensor perceiving and multi-task learning, i.e., multi-modality multi-task (MMMT) models, resulting in diverse Deep Neural Networks (DNNs) with associated DNN layers. For the system, as the diverse DNN layers largely increase the heterogeneity of computing and dataflow patterns, heterogeneous computing becomes a promising solution to address the computation efficiency. it becomes prevailing to integrate dedicated acceleration components such as CPU, GPU, ASIC, and FPGA accelerators into one system to improve overall efficiency. It thus introduces a new problem, heterogeneous model to heterogeneous system mapping (H2H), in which both computation and communication efficiency need to be considered. this dissertation proposes three aspects to enable an efficient heterogeneous model to heterogeneous system mapping. First, a Convolution accelerator design exploration based on FPGA is proposed to address the efficiency in CNN models. Second, the accelerator architecture exploration for LSTM and Transformer are proposed based on FPGA to address the computing efficiency for LSTM and Transformer based models. Third, aiming at system-level formulation, simulation, and optimization, a computation and communication aware heterogeneous model to heterogeneous system mapping algorithm with its associated system-level simulator is proposed. In solving the H2H system mapping, we take the multi-FPGA system as the multi-accelerator platform due to its flexibility in accelerator architecture exploration and fast prototyping procedure