Active Collaborative Planning and Sensing in Human-Robot Teams

As they become increasing capable, autonomous robots can further benefit from human-provided semantic characterizations of uncertain task environments and states. However, the development of integrated strategies which let robots model, communicate, and act on such `soft data' remains challenging. This work presents a framework for active semantic sensing and planning in human-robot teams which addresses these gaps by developing algorithms and techniques to allow formally integrated semantic sensing and planning in human robot teams, leveraging advances in POMDP approximation, multi-modal semantic interaction, and Bayesian data fusion. This approach lets humans opportunistically impose model structure and dynamically extend the range of semantic soft data in uncertain environments, which otherwise yield little information to a lone robot. It also lets robots actively query humans for new semantic data which update understanding and beliefs of unknown environments for improved online planning. Dynamic target search simulations show that active collaborative semantic sensing leads to significant improvements in time and belief state estimates required for interception versus conventional planning, which relies on robotic sensing only. This thesis contains several contributions advancing the state of the art in human-robot collaborative planning. Chapter 3 derives and implements the VB-POMDP algorithm, which provides for continuous state POMDP planning under hybrid continuous-to-discrete semantic sensor observations modeled by softmax functions. Compared to previous methods, this algorithm scales construction of observation models to previously unreachable integrated planning and control problems. Chapter 4 scales the work of Chapter 3 even further, applying a hierarchical framework for the efficient solution of POMDPs in complex continuous state spaces. It also incorporates active human sensing into such problems, using a semantic dictionary of potential robotic queries to facilitate human-robot information transfer. Finally, Chapter 5 extends collaborative human-robot target search to unknown a priori environments, and presents a novel sketch-based approach to multi-level active semantic sensing which allows the transfer of both model and state information without the need for retraining