Computer Vision-Based Automatic Railroad Crossing Monitoring and Track Inspection

Currently, there are many imminent challenges in the railroad infrastructure system of the United States, impacting the operation, safety, and management of railroad transportation. In this work, three major challenges which are overcrowded traffic congestion at the grade crossing, low-efficiency and accuracy on inspection of missing or broken rail track components, and dense rail surface defects without quantification, respectively are studied. The congested railroad grade crossing not only introduces significant traffic delays to travelers but also brings potential safety concerns to the first responders. However, limited studies have been devoted on developing an intelligent traffic monitoring system which is significant to deliver real-time information to the travelers and the first responders to improve the traffic operation and safety at the railroad grade crossing. Except to improve the railroad safety related with travelers and the first responders in the first half, the rest of this dissertation focuses on the track safety related to railroad track components and surface defects. The missing or broken components such as spikes, clips, and tie plates can endanger the safety and operation of railroads. Even though various types of inspection approaches such as ground penetrating radar, laser, and LiDAR have been implemented, the operation needs rich experience and extensive training. Meanwhile, track inspections still heavily rely on manual inspection which is low-accurate, low-efficient, and highly subjective. Moreover, rail surface defects negatively impact riding comfort, operational safety, and could even lead to train derailments. During the past decades, there have been many efforts to detect rail surface defects. Unfortunately, previous approaches for detecting and quantifying of rail surface defects are also limited by the high requirements of specialized equipment and personnel training. The main focus of this work is to design and develop computer vision models to address the technical and practical challenges mentioned above. To cope with each challenge, different models including the object detection model, the instance segmentation model, and the semantic segmentation model have been successfully designed and developed. To train, validate, and test different models, three customized image datasets based on the traffic videos at the grade crossing, railroad component images, and dense rail surface defects images have been built. Specifically, a dense traffic detection net (DTDNet) is developed integrating the Transformer Attention (TA) module for better modeling of global context information and the learning-to-match detection head for optimizing object detection and localization using a likelihood probability fashion. A unique grade crossing traffic image dataset including congested and normal traffic during both daytime and nighttime is established. The proposed DTDNet and other state-of-the-art (SOTA) models have been trained, tested, and compared. The proposed DTDNet outperforms other SOTA models in the test cases. Regarding the automatic track components inspection, the real-time instance segmentation model and the YOLOv4-hybrid model have been designed, trained, tested, and evaluated. The first public rail components image database has been built and released online. Compared to the original YOLACT model and the Mask R-CNN model, the training performance has been improved with the improved instance segmentation model. The detection accuracy on the bounding box and the mask has been improved and the inference speed can achieve the real-time speed. With respect to the YOLOv4-hybrid model, it outperforms other SOTA models on the training performance and the field tests with missing or fake rail track components. As for the rail surface defect inspection and quantification, the optimized Mask R-CNN model and the newly proposed lightweight Deeplabv3Plus model using Lovász-Softmax loss (LDL model) have been trained, tested, evaluated, and compared on our rail surface defects image database. Experimental results confirm the robustness and superiority of our model on defect segmentation. Besides, an algorithm is proposed to quantify rail surface defect severities at different levels using our rail surface defects image data. Overall, this dissertation helps to improve the railroad safety by developing and implementing advanced computer vision-based models for better tracking monitoring and inspections