Visuelle Erkennung von Handposituren für einen interaktiven Gebärdensprachtutor

This thesis presents an interactive sign language tutor that visually verifies the user's sign production. Similar to a human teacher, the system offers corrective feedback in case of mistakes or deviations, thereby helping the learner perform the sign correctly. To support hand posture recognition the user wears colored cotton gloves that do not constrict hand motion. Images are recorded using a monocular camera in an uncontrolled environment. The color markers' two-dimensional geometric features form the basis for subsequent hand posture reconstruction. This step is the work's primary focus. From a precise mathematical problem description a new concept is developed which explicitly considers ambiguities inherent in the input data and is able to resolve them by exploiting temporal correlation in the signing process. At the core of the approach is a 3D hand model that accurately simulates the functional anatomy of the human hand. This model has the same color markers as the cotton glove worn by the user and therefore allows computation of the same feature set. The recognition task thus consists in finding joint and viewing angles that match the model's features to the extracted features. To this end, a similarity measure based on the Hausdorff distance has been designed. Sign verification considers manual features, i.e. location, speed of movement, handshape, and orientation. Different sign phases can be modeled by weighting these components accordingly. By evaluating linguistic criteria the sign assessment is comprehensible to the learner. In case of a signing mistake both a textual message and a visual correction are displayed. The hand model is shown moving from the incorrect posture to the correct one, pointing out the differences between the two. This demonstrative feedback is intuitively understandable and can be compared to that which a human teacher might give. To quantify recognition accuracy an extensive evaluation was carried out. The average error, measured by the distance of corresponding finger tips in actual and recognized posture, is as low as 1.53 cm. This approaches the accuracy achievable by a human observer and allows accurate verification of the recorded sign. An interactive sign language tutor constitutes an innovative application of visual hand posture recognition. Extraction of all manual features without restricting handshape or orientation in any way is a challenging problem. The method developed in this thesis successfully solves this task