Real Time Neural Network-based Face Tracker for VR Displays

Tracking technology for Virtual Reality (VR) applications typically requires the user to wear head-mounted sensors with transmitters or wires. This paper describes a video-based, real-time, low-latency, high-precision 3D face tracker specifically designed for VR displays that requires no sensors, markers, transmitters, or wires to be worn. One center camera finds the 2D face position using Artificial Neural Networks (NN) and recognizes and tracks upright, tilted, frontal and non-frontal faces within visually cluttered environments. Two more left and right (L/R) cameras obtain the 3D head coordinates using a standard stereopsis technique. This paper presents a novel idea to train the NN to a new face in less than two minutes, and includes background training to avoid recognition of false positives. The system utilizes diffuse infrared (IR) illumination to avoid computing time-consuming image normalization, to reduce illumination variations caused by the physical surroundings, and to prevent visible illumination from interfering with the user’s field of view