Content driven energy efficiency analysis of hardware accelerated spatial filters for digital image processing

Portable and mobile computing devices that run on batteries such as mobile phones, low-power and ultra-low-power IoT devices, wearable computing devices, wireless video sensor nodes etc. have become an indispensable part of human daily life. One of the major challenges is to have the longest battery life in such devices. Battery life time of the device depends on its energy efficiency which depends on the power consumption of the device. Power consumption of a device can be derived from the rate at which the device consumes energy. Many of these modern devices have touch screens where the content of the applications running in the devices are displayed. Applications running on these devices are dominant in multimedia content including video and images. It has been established that processing image and video content consume more power than any other content. Therefore, these applications involving image and video processing are the major factors in reducing the battery lifetime of such devices. Moreover, the demand on the computing performance of these portable devices is ever increasing. The battery technology and advances in power efficiency of silicon chips used in these devices considerably lag behind the performance enhancements deployed in these portable devices. Energy efficiency optimisation has become an essential objective in the design of modern embedded systems. It is important to mention the three key prevailing technological bottlenecks for high performance computational efficiency gains. These are the memory bottleneck, the Instructional Level Parallelism (ILP) bottleneck and the power bottleneck. The main motivation of this thesis is to address the third bottleneck, the power bottleneck. At the same time, most of the modern portable computing devices contain one or more of System-On-Chip (SoC) type integrated circuits within which there are more than one hardware accelerators implemented to perform dedicated computing functions. The fundamental blocks within the logic circuit are logic gates which are realised from transistors. Dynamic power consumption of a digital logic circuit is directly proportional to the switching of the transistors in the circuit. There are two main factors that affect the switching: the clock frequency and the variation in the input stimulus at the inputs of the circuit. In order to understand the decomposition of the contribution to the power consumption of such a device from the image and video content perspective, it is important to quantify what constitutes the content of a digital image or video. The next step is to find out how these constituent elements impact the power consumption of the device. Digital images are comprised of pixels and these pixels are samples of intensity values represented in binary numbers i.e. 1s and 0s. The variation in the content is represented by the variation in the values of these pixels and vice versa. Image processing mainly involves performing operations on images i.e. in two dimensions. At the sub-symbolic level, the mathematical operations (i.e. convolution operations consisting of Multiplication and Addition, MAC) need to be repeated on the image data numerous times. Accordingly, it remains difficult to achieve real-time performance in software-based implementations of image processing. Therefore, in the modern devices, these operations are accelerated in the bespoke digital circuits in the form of hardware accelerators within the SoCs. In this thesis, the impact of image content on the power consumption and energy efficiency of the most commonly used hardware accelerator architecture for image processing, a spatial filter, is investigated. The first step is to quantify the content of an image into varying frequency sinusoidal gratings of different orientation, phase and contrast. This concept is mainly used in the fields of optometry and biological vision. These areas are combined into this research work by creating a dataset of synthetic images whereby the spatial frequency, phase, contrast and orientation of the sinusoidal grating images are controlled to investigate its impact. A semi-automated experimental framework with a configurable library of hardware accelerated spatial filters is developed in order to explore the impact. Multivariate regression is performed on the selected independent variables for the dependent variable and a statistical model is derived that enables estimation of the energy efficiency of hardware accelerated spatial filters. The model enables algorithm designers to explore and evaluate the energy efficiency of their algorithms without actually implementing them into hardware. The model facilitates the design space exploration of spatial filters with energy efficiency as the objective. The results show that even a featureless plain grey image consumes dynamic power when processed in a digital circuit. This is mainly because of the inherent switching present due to the pixels represented in the binary number format. Moreover, the impact of contrast and phase in an image of sinusoidal grating image is not statistically significant. Maximum amount of energy is consumed when the orientation of the sinusoidal grating in the image is at 0 degrees and the least energy is consumed when the orientation is at 90 degrees. This is due to the row-order scanning of the image and the horizontal symmetry of the hardware blocks to store the image rows. The results show that there is statistically significant impact on the energy efficiency of the spatial filter for varying orientation and spatial frequencies present in the image. The energy efficiency of the template spatial filter drops to 12.5% for the maximum spatial frequency 64 cycles per image and O-degree orientation for a given image of a two-dimensional sinusoidal grating of size 128x 128 pixels whereas, for an orientation of 90 degrees, the energy efficiency is 72.38%. This research provides new insights into the impact of an image content on the power and energy consumption of the hardware accelerated image processing function employed to process the image. The findings presented in this thesis contribute new knowledge regarding the impact of image granularities on processing energy consumption and as such these findings will help inform strategies for energy efficiency gains in image processing