Algorithm-Hardware Co-Optimization for Cost-Efficient ML-based ISP Accelerator

In this paper, we present an advanced algorithm-hardware co-optimization method for designing an efficient accelerator architecture for image signal processing (ISP) with deep neural networks (DNNs). Based on the systolic-array structure, for performing the target network model, we newly introduce two evaluation metrics, each of which is dedicated to fairly representing either the processing speed or the energy consumption. Then, the overall evaluation metric is defined to test each systolic array, finding the initial array configuration for the given number of total multipliers. From the initial array, several array-scaling methods are then presented to find the most cost-efficient array structure. In addition, the original ML model is adjusted to further enhance the overall efficiency with subtle quality drops of image outputs. Implementation results in 28nm CMOS technology show that the proposed co-optimization method successfully finds the cost-efficient systolic accelerator architecture for ISP applications, improving the energy efficiency by 51% compared to the straightforward array design.1