Tracking facial features under occlusions and recognizing facial expressions in sign language

This paper presents work towards recognizing facial ex-pressions that are used in sign language recognition. Facial features are tracked to effectively capture temporal visual cues on the signer’s face during signing. A Bayesian frame-work is proposed as a feedback mechanism to the Kanade-Lucas-Tomasi (KLT) tracker for reliably tracking facial fea-tures in the presence of head motions and temporary occlu-sions by hand. This mechanism relies on a set of face shape subspaces learned by Probabilistic Principal Component Analysis with an update scheme to adapt to persons with different face shapes. The results show that the proposed tracker can track facial features with large head motions, substantial facial deformations, and temporary facial oc-clusions by hand. The tracked results were input to a recog-nition system comprising HMMs and a NN to recognize four common American Sign Language facial expressions. 1