Seamless bare-hand interaction in mixed reality

Unrealistic Mixed Reality (MR) experiences can be caused by unprocessed occlusions between real and augmented objects during interactions. Depth knowledge is indeed key to achieve a seamless visualisation when a bare hand interacts with an augmented object. This can be addressed by blending real-time 3D finger tracking information with the visualisation of the hand in MR. We propose an approach that automatically localises the hand in RGB images and associates the respective depth; estimated with an auxiliary infrared stereo camera used for hand tracking; to each RGB hand pixel. Because misalignments between the outline of the hand and its depth may occur due to tracking errors; we use the distance transform algorithm to densely associate depth values to hand pixels. In this way hand and augmented object depths can be compared; and the object can be rendered accordingly. We evaluate our approach by using an MR setup composed of a smartphone and a Leap Motion mounted on it. We analyse several hand configurations and measure the erroneously classified pixels when occlusions occur