Efficient Video Processing on Embedded GPU

Learn how to develop and test a 4K video processing and streaming application on the NVIDIA Jetson TX1/TX2 embedded platform with GStreamer. To achieve real-time video processing, the diverse processing resources of this high-performance embedded architecture need to be employed optimally. The heterogeneous system architecture allows capturing, processing, and streaming of video with a single chip. The main challenges lie in the optimal utilization of the different hardware resources of the Jetson TX1 (CPU, GPU, dedicated hardware blocks) and in the extensive software stack (from Linux drivers to GStreamer application). We will discuss variants, identify bottlenecks, and show the interaction between hardware and software. Simple capturing and displaying video from the built-in camera can be achieved using out-of-the-box methods. However, for capturing 4K from HDMI we had to dig into the documentation, rewrite the drivers for the Video capture system, write a driver for an external HDMI to CSI converter and figure out efficient zero-copy methods. GPU-based enhancements were developed and integrated for real-time video processing tasks (scaling and video mixing)