Hand Gesture Recognition for Smartphone-Based Augmented Reality Applications

Hand Gesture Recognition (HGR) is a principal input method in head-mounted Augmented Reality (AR) systems such as HoloLens, but the high cost and limited availability of such systems prevent HGR from becoming more prevalent. Alternatively, smartphones can be used to provide AR experiences, but current smartphones were not designed with HGR in mind, making development of HGR applications more challenging. This study develops a software-based framework that implements HGR as a principal input method for smartphone AR applications. This framework assumes a contemporary smartphone with dual back-facing cameras, which enable stereo imaging and thus allow extraction of limited depth information from the environment. Several image processing techniques, derived and improved from previous work, were used to filter the noisy depth information to segment the user’s hand from the rest of the environment, and then to extract the pose of the hand and fingers in real-time. The framework additionally facilitates the development of cross-platform AR applications for both head-mounted (HoloLens) and smartphone configurations. A user experiment is held to determine whether a smartphone-based AR application developed using our HGR framework is comparable in usability to the same application on the HoloLens. For each device, participants were asked to use the application and fill out a usability questionnaire. They were also asked to compare the two systems at the end. This experiment shows that, despite the current limitations of smartphone-based HGR, the smartphone system’s usability is competitive with that of the HoloLens. This study ends with recommendations for future development