Semantic segmentation of roof materials in urban environment by utilizing hyperspectral and LiDAR data

Mapping areas in an urban environment can be challenging due to various materials and manufactured structures. The urban environment is a mix of natural and artificial materials, and finding the right object of a specific material is a challenge even for the trained eye. Therefore, by applying high spectral resolution hyperspectral imagery it is possible to examine surface materials based on spectral signature. Combined with LiDAR, it is also feasible to detect the geometrical structure of the surface. These data can be exposed to a machine learning algorithm to recognize objects automatically. In this study machine learning algorithms are exposed to airborne images of roof materials. This thesis presents an application of semantic segmentation for roof materials based on fused hyperspectral (HySpex VNIR-1800 and SWIR-384) and LiDAR (Riegl VQ-560i) data acquired from 2021 over Bærum municipality near Oslo in Norway. The machine learning algorithm is a semantic segmentation model named Res-U-net with a U-net architecture and a ResNet34 backbone. The Res-U-Net is a supervised neural network with high capacity to learn high-dimensional airborne data. The model returns a mask of the urban area that pinpoints the roofs’ position and materials. The ground truth is generated with information from field work, a geographical database and the watershed algorithm for object detection. This ground truth consists of nine different roof materials and background. The semantic segmentation model is optimized by testing different model configurations for this specific problem. The best model scores 0.903, 0.896, and 0.579 in accuracy score, F1 score weighted and Matthews Correlation Coefficient. For the binary problem of detecting roof the model scores 0.948, 0.946, and 0.767 on the same metrics. This study demonstrates that semantic segmentation is viable for localizing and classifying roof materials with fused hyperspectral and LiDAR data. Such an analysis can potentially automate several mapping chores and manual assignments by systemically processing a larger area in a short time to free human capacity