Grasp Recognition and Manipulation with the

Summary. We describe a novel user interface for natural, whole hand interaction with 3D environments. Our interface uses a graspable device called the Tango, which looks like a ball but measures contact pressures on its surface at 256 tactual elements (taxels) at a high rate (100 Hz). The acceleration of the device is also measured. The key idea is to use this information to recognize the shape and movement of the user’s hand grasping the object. This allows the user to interact with 3D virtual objects using a hand avatar. The interface provides passive force feedback, and is easier to use than interfaces that require wearing gloves or other sensors on the hand. We describe a rotationally invariant matching algorithm for recognizing the hand shape from examples of previous interaction collected with motion capture. We also describe examples of 3D interaction using our system.