Visual Tracking of Human Hand and Head Movements and Its Applications

Tracking of human body movements is an important problem in computer vision with applications in visual surveillance and human-computer interaction. Tracking of a single hand moving in space is addressed and a set of applications in human computer interaction are presented. In this approach, a disparity map and motion fields extracted from a stereo camera set are modelled using a robust estimation method. Then, the absolute position and orientation of the hand in space are estimated and the central region of the hand is tracked over time. Virtual drawing in space, a virtual marble game, and 3D object construction are shown as the applications of the single hand tracking. Algorithms are presented for tracking the hands and head of a person or several interacting people viewed by a set of cameras in 3D. The problem is first defined as a general multiple object tracking problem in a multiple sensor environment and a two layered solution is proposed. The proposed solution includes a low level particle filtering layer to track individual targets in parallel, and a finite state machine to analyze the interactions between the targets and apply application specific heuristics. A set of activity recognition experiments in visual surveillance show the usefulness of the system. The recognized activities involve interactions between the hands and head of people and objects. A color analysis scheme and a technique for combining information from different cameras are presented. They are used to detect carried objects and exchanges between the hands