An fpga-based stream processor for embedded real-time vision with convolutional networks

Many recent visual recognition systems can be seen as being composed of multiple layers of convolutional filter banks, interspersed with various types of non-linearities. This includes Convolutional Networks, HMAX-type archi-tectures, as well as systems based on dense SIFT features or Histogram of Gradients. This paper describes a highly-compact and low power embedded system that can run such vision systems at very high speed. A custom board built around a Xilinx Virtex-4 FPGA was built and tested. It mea-sures 70 × 80 mm, and the complete system—FPGA, cam-era, memory chips, flash—consumes 15 watts in peak, and is capable of more than 4 × 109 multiply-accumulate oper-ations per second in real vision application. This enables real-time implementations of object detection, object recog-nition, and vision-based navigation algorithms in small-size robots, micro-UAVs, and hand-held devices. Real-time face detection is demonstrated, with speeds of 10 frames per sec-ond at VGA resolution. 1