Material Detection with Thermal Imaging and Computer Vision: Potentials and Limitations

The goal of my masters thesis research is to develop an affordable and mobile infraredbased environmental sensoring system for the control of a servo motor based on material identification. While this sensing could be oriented towards different applications, my thesis is particularly interested in material detection due to the wide range of possible applications in mechanical engineering. Material detection using a thermal mobile camera could be used in manufacturing, recycling or autonomous robotics. For my research, the application that will be focused on is using this material detection to control a servo motor by identifying and sending control inputs based on the material in an image. My thesis is driven by the following research question: how does infrared imaging compare to visible light in terms of prediction accuracy both in ideal and non-ideal scenarios? This question is motivated by the fact that there is a lack of knowledge on the distinction between the qualities of thermal imaging and RGB imaging for computer vision, especially with the use of an affordable mobile camera. To address this gap and answer the research question, this thesis aims to achieve three objectives: 1) to create a dataset and train a thermal imaging convolutional neural network (CNN) for material detection, 2) to create a testbed that will utilize the material detection for the control of an actuator, and 3) to compare the performance of thermal imaging vs. RGB imaging in terms of detection accuracy for both ideal and non-ideal scenarios. To achieve these objectives, a large number of infrared and RGB images must be collected and pre-processed to create a dataset for the training of CNN models and the prediction of material types. A protocol must also be developed to establish the real-time communication between the mobile thermal device and the actuator to relay this material information. An in-depth understanding is gained of the benefits and drawbacks in terms of accuracy’s in ideal and non-ideal scenarios while using an affordable thermal mobile camera as opposed to traditional RGB cameras for material detection. These methods were tested on a small-scale prototype device consisting of a Raspberry Pi and a SG90 servo motor. The way each data type is pre-processed is different, e.g., using dynamic range quantization vs. standardization, in order to obtain the best model performances. Our results show that the thermal imaging model performed better than RGB model in non-ideal scenarios where is was dark (52% average accuracy vs. 46%), but was not able to outperform RGB imaging in ideal scenarios (74% average accuracy vs. 95%). While this conclusion is not surprising and falls in our expectation, the quantification of the differences between RGB imaging and thermal imaging for material detection and the systematic approach developed are the new knowledge generated. It reveals the potentials and limitations of infrared image-based computer vision and therefore sets the foundation for future work with thermal imaging as it relates to environmental sensing, autonomous applications, and under what conditions this application can be made