Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging

Background: A smartphone is a promising tool for daily cardiovascular measurement and mental stress monitoring. A smartphone camera-based PhotoPlethysmoGraphy (PPG) and a low-cost thermal camera can be used to create cheap, convenient and mobile monitoring systems. However, to ensure reliable monitoring results, a person has to remain still for several minutes while a measurement is being taken. This is very cumbersome and makes its use in real-life mobile situations quite impractical. // Objective: We propose a system which combines PPG and thermography with the aim of improving cardiovascular signal quality and capturing stress responses quickly. // Methods: Using a smartphone camera with a low cost thermal camera added on, we built a novel system which continuously and reliably measures two different types of cardiovascular events: i) blood volume pulse and ii) vasoconstriction/dilation-induced temperature changes of the nose tip. 17 healthy participants, involved in a series of stress-inducing mental workload tasks, measured their physiological responses to stressors over a short window of time (20 seconds) immediately after each task. Participants reported their level of perceived mental stress using a 10-cm Visual Analogue Scale (VAS). We used normalized K-means clustering to reduce interpersonal differences in the self-reported ratings. For the instant stress inference task, we built novel low-level feature sets representing variability of cardiovascular patterns. We then used the automatic feature learning capability of artificial Neural Networks (NN) to improve the mapping between the extracted set of features and the self-reported ratings. We compared our proposed method with existing hand-engineered features-based machine learning methods. // Results: First, we found that the measured PPG signals presented high quality cardiac cyclic information (relative power Signal Quality Index, pSQI: M=0.755, SD=0.068). We also found that the measured thermal changes of the nose tip presented high quality breathing cyclic information and filtering helped extract vasoconstriction/dilation-induced patterns with fewer respiratory effects (respiratory pSQI: from M=0.714 to M=0.157). Second, we found low correlations between the self-reported stress scores and the existing metrics of the two cardiovascular signals (i.e. heart rate variability and thermal directionality metrics) from short measurements, suggesting they were not very dependent upon one another. Third, we tested the performance of the instant perceived stress inference method. The proposed method achieved significantly higher accuracies than existing pre-crafted features based-methods. In addition, the 17-fold Leave-One-Subject-Out (LOSO) cross-validation results showed that combination of both modalities produced higher accuracy in comparison with the use of PPG or thermal imaging only (PPG+Thermal: 78.33%; PPG: 68.53%; Thermal: 58.82%). The multimodal results are comparable to the state-of-the-art automatic stress recognition methods that require long term measurements (usually, at least a period of 2 minutes is required for an accuracy of around 80% from LOSO). Lastly, we explored effects of different widely-used data labeling strategies on the sensitivity of our inference methods. Our results showed the need for separation of and normalization between individual data. // Conclusions: Results demonstrate the feasibility of using smartphone-based imaging for instant mental stress recognition. Given that this approach does not need long-term measurements requiring attention and reduced mobility, we believe it is more suitable for mobile mental healthcare solutions in the wild