Improving Accuracy of Human Activity Recognition Using Adaptive Weighted Fusion Based on Skeleton Data and Image Information

Human Activity Recognition is to make a computer recognize human activity using data collected from video camera or devices such as Kinect. There has been active research on activity recognition models using skeleton data, which is less sensitive to background information and angle, and the overall accuracy of the models is improving year by year. However, some activities such as “reading” and “writing” are difficult to recognize, and the accuracy has not improved much. Therefore, in this research, for activities that are difficult to recognize with skeleton data alone, the image information including appearance information and objects used by the subject are fused. In addition, unlike existing fusion methods, we proposed a method that fuses the output of each deep learning model with different weights depending on the output results. The weights are predicted by a machine learning model that has been previously trained with weights calculated by the newton method. The experimental results show that the accuracy of difficult-to-recognize activities can be improved by fusing image information using the proposed fusion method. It is also shown that the overall accuracy of the proposed fusion method in this research is higher than the existing fusion methods