Automatic recognition of whole-of-body gestures distilled via a user-centric design approach

Gestures are appealing as a natural and intuitive form of Human Computer Interaction (HCI). This thesis presents an empirical evaluation of whole-of-body gesture design and recognition. A user-centric design experiment precedes the application and evaluation of machine learning techniques.Initially, user preferred whole-of-body gestures to control an avatar in a high fidelity 3D virtual environment were collected through a participatory design study. Thereafter, in order to identify archetypal gesture patterns, collected designs were codified using a formal notation based on Laban Movement Analysis. Subsequently, a clustering technique was applied and a user-preferred gesture library was deduced. The automatic recognition of the deduced gestures was explored. In particular, recognition of pre-segmented gestures, continuous gestures without extraneous gestures (those not supposed to be interpreted by the system) and continuous gestures including extraneous gestures were considered. Recognition of the pre-segmented gestures were explored using four variations of a Dynamic Time Warping (DTW) algorithm (including a novel approach proposed herein) and a network of continuous Hidden Markov Models (HMM).The parameters that impact the classification accuracy of these algorithms were explored. These include: two data sources (vision based and accelerometers); five gesture representations (linear velocity, angular velocity, acceleration and their combinations); three joint configurations; amount of training data; size of the K-nearest neighbourhood used in DTW decision making; two similarity measures for the DTW algorithms (Euclidean and Mahalanobis distance); and the number of states in HMM. Based on the pre-segmented gesture recognition experiments findings, HMM was employed with its optimal parameters when conducting the continuous gesture recognition experiments.Key findings presented within the thesis include: 1) a high diversity exists in user preferred whole-of-body gestures, but the application of labelling and automated hierarchical clustering allow archetypal gesture designs to be captured. 2) The proposed DTW algorithm and HMM adapted for whole-of-body gesture recognition achieved better recognition accuracies and required less training data than the state-of-the-art DTW when recognising pre-segmented whole-of-body gestures. 3) Whole-of-body gestures including extraneous gestures can effectively recognise by the selected control system when the system is trained with proper data