Computer Vision Based Automated Monitoring and Analysis of Excavation Productivity on Construction Sites

Construction equipment is the main component of construction production, and the equipment costs are usually one of the constructor’s biggest expenditures. The productivity of construction equipment plays an important role in completing construction projects within schedule and under budget. Therefore, effectively monitoring equipment productivity is critical to improve construction productivity and control the cost. In order to monitor the productivity of the equipment, numerous research works are focusing on monitoring the operation process of the equipment. Through the continuous monitoring of equipment operations, detailed information about the productivity and performance indicators, such as activities, idling time, and work cycles can be estimated. In recent years, different information technologies, such as machine learning and Real-time Location Systems (RTLS), have been used for equipment productivity monitoring. Based on the type of the data collected, technologies can be categorized into computer vision (CV)-based methods and sensor-based methods. The CV-based methods collect equipment operation data from site surveillance cameras as videos or images. Sensor-based methods install sensors and/or tags (such as Radio Frequency Identification (RFID), Global Positioning System (GPS), Ultra-wideband (UWB), Inertial Measurement Unit (IMU), etc.) on the equipment and the construction site to collect the position and pose information of the equipment. Accordingly, the work states and activities of the equipment are identified by analyzing the data collected from cameras or sensors. The location and trajectory data collected from the sensors can be used to directly estimate the activity of the equipment. Finally, based on the activity information, the productivity of the equipment can be estimated in the form of equipment operation time or soil quantity. However, the previous studies did not consider the activity recognition and productivity analysis of multiple excavators. Moreover, the reasons that cause equipment low productivity, such as idling reasons, have not been considered in previous CV methods. Furthermore, the sensor-based methods can be expensive and impractical to use for the large fleet of construction equipment. Compared with the sensor-based methods, CV-based methods are growing and becoming more efficient. This research aims to monitor the productivity of the excavator using a CV-based method and has the following objectives: (1) Comparing the CV-based and sensor-based methods of earthmoving equipment productivity monitoring to identify the advantages and limitations of each approach, and proposing a roadmap for the future work directions of automated equipment productivity monitoring; (2) Developing an end-to-end CV-based method for automatically recognizing activities of multiple excavators; (3) Developing a framework to analyze the productivity of the excavator based on the activity recognition results; and (4) Improving the earthmoving productivity by identifying the idling reasons of excavators and trucks. A systematic literature review is conducted to cover the recent studies in the area of activity recognition and to compare the CV-based methods with the sensor-based methods for earthmoving equipment productivity monitoring. Then, a roadmap is proposed to suggest future research directions for automatic equipment productivity monitoring. Accordingly, a framework that integrates three Convolutional Neural Networks (CNNs) is proposed for automatic detection, tracking, activity recognition and productivity analysis of excavators. The proposed framework has been tested with the videos recorded from real construction sites. The overall activity recognition has achieved 87.6% accuracy. The productivity calculation has achieved 83% accuracy, which indicates the feasibility of the proposed framework for automating the monitoring of excavator’s productivity. Furthermore, knowing that idling is one of the main reasons for low productivity, a CV-based method was proposed to identify the idling reasons in earthmoving operations by analyzing the workgroup and interactive work states of the excavators and trucks. In this method, the activities of the excavators and trucks were identified using CNNs. Then, the onsite camera was calibrated to estimate the proximity and identify workgroups of excavators and trucks. By calculating the relationships between each excavator and the surrounding truck(s), the potential reason for idling can be identified. The proposed method has been validated with videos from several construction sites showing promising results. The contributions of the research are: (1) Proposing a roadmap to show the future research directions of automatic equipment productivity monitoring; (2) Proposing a CV-based method to recognize the activities of multiple excavators with the state-of-the-art three dimensional (3D) CNN; (3) Proposing a framework that integrates the detection, tracking, and activity recognition techniques to monitor the operation process and analyze the productivity of the excavator; and (4) Developing a method to automatically identify the idling reasons of excavators and trucks based on their interactive work states, which contributes to a better understanding of earthmoving productivity under the dynamic and complex construction site conditions