CloudUP—Upsampling Vibrant Color Point Clouds Using Multi-Scale Spatial Attention

In recent years, there has been a noticeable increase in the inclination towards digitizing our surroundings, encompassing various domains such as virtual reality, cultural heritage conservation, and architectural representation. The computation of high-resolution three-dimensional (3D) colored point clouds and meshes holds significant importance for such applications. However, traditional structure-from-motion (SfM) techniques may produce sparse 3D point clouds when low-resolution input images are used, resulting in a low-quality mesh generation. Traditional point cloud upsampling techniques that improve the 3D point cloud resolution typically work on LiDAR-generated point clouds devoid of color information. Furthermore, most learned point cloud upsampling techniques compute graph features that capture local information by identifying a local neighborhood in a limited region around a point and hence may result in sub-optimal representation. To address these limitations, we propose CloudUP, a colored 3D point cloud upsampling approach that utilizes multi-scale spatial attention. Specifically, we design a novel Multi-Scale Point-Cloud Feature Extractor (MPFE) by employing attention across the scales to extract point cloud features and effectively capture 3D shape information of the points relative to its neighborhood. We further extract spatial neighborhood-guided color features used to predict the color for the upsampled points. The color prediction is trained with a content-preserving loss function that aims to maintain intricate details and vivid colors. Our color refinement pipeline is guided by a vibrant colored dataset (collected by us) to assist in preserving the 3D contents