Enhancing Point Cloud Density with Stereo Images

3D object detection is a critical perception task in self-driving cars to ensure safetyduring operations. In order to perceive visual contexts of their surroundings, autonomous vehicles utilize many sensors to accurately detect not only the object’s location but also what kind of object it is. Many vehicles now rely heavily on a light detection and ranging(LiDAR) sensor to measure the precise distance between a car and the objects. 3D object detection models may under-perform because of the sparse nature of LiDAR point clouds.In order to improve performance of existing 3D object detection models, we developed a method to enhance point cloud density with the aid of stereo images. We trained a depth estimation network using stereo images to produce pseudo-LiDAR point clouds. Then, we enhanced point cloud density by fusing a regular LiDAR point cloud with a post-processed pseudo-LiDAR point cloud. After training separate 3D object detectors with different types of input point cloud, our proposed approach on enhancing point clouds performed the best. When compared to 3D object detection results on regular LiDAR point clouds, with a minimum IoU threshold 0.7, our approach increased 24.7 average precision (AP) on easy difficulty objects and 16.5 on moderate difficulty objects on 3D bounding box evaluations. Also, with the same IoU threshold, APs for Bird-Eye-View (BEV) evaluations increased by 14.1 on easy difficulty objects and 7.9 moderate difficulty objects. After careful statistical and visual evaluations, we concluded that our method of enhancing point clouds could improve existing 3D object detection performance