3D hand pose reconstruction using specialized mappings

A system for recovering 3D hand pose from monocu-lar color sequences is proposed. The system employs a non-linear supervised learning framework, the specialized mappings architecture (SMA), to map image features to likely 3D hand poses. The SMA’s fundamental components are a set of specialized forward mapping functions, and a single feedback matching function. The forward functions are estimated directly from training data, which in our case are examples of hand joint configurations and their corre-sponding visual features. The joint angle data in the train-ing set is obtained via a CyberGlove, a glove with 22 sen-sors that monitor the angular motions of the palm and fin-gers. In training, the visual features are generated using a computer graphics module that renders the hand from ar-bitrary viewpoints given the 22 joint angles. The viewpoint is encoded by two real values, therefore 24 real values rep-resent a hand pose. We test our system both on synthetic sequences and on sequences taken with a color camera. The system automatically detects and tracks both hands of the user, calculates the appropriate features, and estimates the 3D hand joint angles and viewpoint from those features. Results are encouraging given the complexity of the task.