Study on Methods of 3-D Gaze Tracking

視線追蹤技術在用來推論使用者所關注的焦點上一直扮演重要的角色。在論文中，我們發展兩個非接觸式三維視線追蹤的方法。一個是基於三維模型的視線追蹤方法；它使用立體視覺技術與高解析度的紅外線影像，適用於有高準確度要求的良好控制環境之下。另一個是以眼睛影像為基礎的視線追蹤方法；它使用市面上一般使用的攝影機且不需要額外的硬體設備，能夠運作在一個較寬鬆的環境但依然能提供一定範圍的準確度視線估測。 論文的第一部份提出一個基於三維模型的視線追蹤方法。角膜反射(或亮點)在三維模型的視線追蹤方法是一個很重要的特徵。然而當角膜反射在大視角範圍的使用下有亮點消失的問題，使得效能和使用的範圍因此受限。雖然瞳孔輪廓特徵提供相輔的資訊來幫助視線方位的計算，但現行的方法中並沒有在瞳孔輪廓特徵上提供一個正確角膜折射模型，因而導致不準確估測結果。我們提出一個以瞳孔輪廓特徵為基礎的視線追蹤方法，它能相容於角膜折射。此外我們也說明以亮點特徵與以瞳孔輪廓為特徵兩者方法的關連性，並將兩個方法整合在一個統一框架中。我們說明整合方法的必要性，結合兩個方法所提供的互補的資訊對於增強系統的穩定性與靈活性是有幫助的。 論文的第二部份發展一個以眼睛影像為基礎視線追蹤方法。直接使用眼睛影像作為基礎的視線追蹤方法的困難點在於攝影機所觀察到眼睛影像會隨著頭部方位而有所改變。為了克服這個問題，我們提出一個結合三維頭部方位追蹤與眼睛影像為基礎的視線追蹤方法。我們提出一個以三維人臉模型的頭部方位追蹤技術；它使用眼睛位置來做為引導輔助頭部追蹤，使得系統具有更穩固的特性；並發展一套自動初始化、追蹤失敗重新初始化的機制。我們基於霍夫隨機森林演算法在頭部方位空間與影像眼睛外觀的聯合空間建構出相鄰結構，然後在相鄰結構樣本中利用 l1 回歸法來找出最佳的視線估測。所建構的系統在使用者在頭部移動下能仍運作並提供一定準確度的三維視線估測。Gaze estimation techniques play important roles in inferring the focus of attention all the time. In this dissertation, we develop two remote 3-D gaze tracking methods. One is a 3-D model-based method that uses stereo vision with high resolution infrared imaging, which satisfies the high accuracy requirement for the use in controlled environments; the other is an appearance-based method that uses a commercial-off-the-shelf camera without requiring any dedicated hardware, which provides moderate estimation accuracy but can be used in less constrained environments. The first part of this thesis describes a 3-D model-based method for gaze tracking. Corneal reflection (or glint) is an important feature used by many 3-D model-based methods. However, when the operation range of a gaze tracking system is enlarged, glint feature-based (GFB) approaches will suffer from performance mainly due to the missing glint problem. Although the pupil contour feature may provide complementary information to help estimating the eye gaze, existing methods do not properly handle the cornea refraction problem which leads to inaccurate results. We describes a contour-feature based (CFB) 3-D gaze tracking method that is compatible to cornea refraction. We also show that both the GFB approach and the CFB approach can be formulated in a unified framework and, thus, they can be easily integrated. Furthermore, it is shown that the integration of the proposed CFB method and the GFB method is necessary because the two methods provide complementary information that helps to leverage the strength of both features and provides robustness and flexibility to the system. In the second part of this thesis, an appearance-based gaze tracking system is presented. The main difficulties for the appearance-based method are that eye appearances are varied with head motion. To overcome the difficulty, we propose a 3-D gaze tracking method combining head pose tracking and appearance-based gaze estimation. We present a robust model-based head pose tracking method guided by the eye location information and develop an automatic mechanism to deal with the problems of pose initialization, tracking loss recovery and re-initialization. We use a random forest approach to model the neighbor structure of the joint head pose and eye appearance space. l1-optimization is then used to seek for the best solution for regression from the selected neighboring samples. The constructed system is capable of analyzing the 3-D visual LoS of a person allowing the movement of his/her head and eye in a more natural manner but still provides moderate estimation accuracy.Abstract vii List of Figures xiii List of Tables xvii 1 Introduction 1 1.1 Background and Motivation . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Human Eye Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Cornea Reflection and Virtual Pupil . . . . . . . . . . . . . . . . . . . . 4 1.4 Overview of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2 Related Work 7 2.1 Feature-based Method . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.1.1 2-D regression-based Approach . . . . . . . . . . . . . . . . . . 8 2.1.2 3-D model-based Approach . . . . . . . . . . . . . . . . . . . . 9 2.2 Appearance-based Method . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.3 Contributions of Our Work . . . . . . . . . . . . . . . . . . . . . . . . . 14 3 3-D Model-based Method 17 3.1 Estimation of the EACC Plane . . . . . . . . . . . . . . . . . . . . . . . 18 3.1.1 Proposed CFB Approach for Estimating the EACC Plane . . . . . 18 3.1.2 Shih and Liu’s GFB Approach for Estimating the EACC Plane . . 24 3.1.3 Singular Cases and the Need for Integration . . . . . . . . . . . . 25 3.2 Estimation of the Line of Sight . . . . . . . . . . . . . . . . . . . . . . . 26 3.2.1 Estimation of the Optical Axis . . . . . . . . . . . . . . . . . . . 26 3.2.2 Inferring the Line of Sight Using the Optical Axis . . . . . . . . . 29 3.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.3.1 Necessity of Using the Contour Feature . . . . . . . . . . . . . . 30 3.3.2 Experiment Setup for Evaluating the Proposed Method . . . . . . 34 3.3.3 Experiments on Synthetic Data . . . . . . . . . . . . . . . . . . . 37 3.3.4 Experiments on Real Data . . . . . . . . . . . . . . . . . . . . . 42 3.4 Summary of the Proposed Model-based Method . . . . . . . . . . . . . . 49 4 Appearance-based Method 51 4.1 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.2 Eye Location Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.2.1 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.2.2 Eye Detection and Localization . . . . . . . . . . . . . . . . . . 56 4.3 Head Pose Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.3.1 Deformable Face Model and Shape-free Facial Representation . . 57 4.3.2 Model Initialization . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.3.3 Head Pose Tracking . . . . . . . . . . . . . . . . . . . . . . . . 61 4.4 Gaze Direction Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.4.1 Neighbor Selection . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.4.2 Gaze Direction Regression . . . . . . . . . . . . . . . . . . . . . 72 4.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 4.5.1 Evaluation of the Head Pose Estimation . . . . . . . . . . . . . . 74 4.5.2 Evaluation of the Gaze Estimation . . . . . . . . . . . . . . . . . 75 4.6 Summary of the Proposed Appearance-based Method . . . . . . . . . . . 78 5 Conclusions 8