Contactless smartphone camera-based heart rate estimation from facial videos for diverse subject skin tones and scenes using synthetic augmentation

The COVID-19 pandemic brought telemedicine applications under the spotlight and the heart rate (HR) is an important clinical vital sign in the evaluation of cardiorespiratory and hemodynamic stability. However, both deep learning- and signal processing-based systems demonstrate biased performance towards dark skin tones in remote HR measurements, as performance measures are restricted to the diversity of dataset subjects. The existing datasets MMSE-HR, AFRL, and UBFC-RPPG contain roughly 10%, 0%, and 5% dark-skinned subjects respectively, leading to poor generalization capability to unseen subjects and lead to unwanted bias toward different demographic groups. We propose a physics-driven algorithm to combat this bias and show a first attempt to overcome the lack of dark-skinned subjects by synthetic augmentation. A joint optimization framework is utilized to translate real videos from light-skinned subjects to dark skin tones while retaining their pulsatile signals. In the experiment, our method exhibits around 31% reduction in mean absolute error for the dark-skinned group and 46% improvement on biasm itigation for all the groups, as compared with previous work trained with just real samples