Interactive Machine Learning for Assistive Robots

Intelligent assistive robots can potentially support elderly persons and caregivers in their everyday lives and facilitate a closer man and machine collaboration as an essential part of the yet to come 5-th industrial revolution. In contrast to classical robotic applications where robots were mostly designed for repetitive tasks, assistive robots will face a variety of different tasks in close contact with everyday users. In particular, it is difficult to foresee the variety of applications beforehand since they depend on a person's individual needs and preferences. This renders preprogramming of all tasks for assistive robots difficult and gives need to explore methods of how robots can learn new tasks at hand during deployment time. Learning from and during direct interaction with humans provides hereby a potentially powerful tool for an assistive robot to acquire new skills and incorporate prior human knowledge during the exploration of novel tasks. Such an interactive learning process can not only help the robot to acquire new skills or profit from human prior knowledge but also facilitates the participation of inexperienced users or coworkers which can lead to a higher acceptance of the robot. However, while on the one hand human presence and assistance can be beneficial during the learning process, on the other hand, close contact with inexperienced users also imposes challenges. In shared workspaces or in close contact with everyday users a robot should be able to adapt learned skills to achieve as little disturbance of humans as possible. It becomes also important to evaluate human preferences about such adaptation strategies, their understanding of interactive learning processes and different ways for human input into learning. To come closer to the goal of intelligent assistive robots is therefore important to develop novel interactive learning methods and evaluate them in different robotic applications. This thesis focusses on three main challenges related to the development of assistive intelligent robots and their interaction with everyday users. The different parts of the thesis contribute not only novel theoretical methods but additionally also evaluations on different robotic tasks with users, that had zero or only little prior experience with robots. The first challenge is to enable robots to learn cooperative skills from a potentially open-ended stream of human demonstrations in an incremental fashion. While learning new skills from human demonstrations has already been exploited in the literature it remains challenging to learn skill libraries from incrementally incoming demonstrations and when the total number of skills is not known beforehand. Therefore, in the first part of the thesis, we introduce an approach for online and incremental learning of a library for collaborative skills. Here, we follow a Mixture of Experts based approach and incrementally learn a library of collaborative skills and a gating model from coupled human-robot trajectories. Once trained, the gating model can decide which skill to choose as an appropriate response to a human motion, based on prior demonstrations and activate the corresponding robot skill. In contrast to existing batch learning methods, our method does not require the total number of skills to be known a priori and can learn new skills as well as update existing skills from multiple human demonstrations. The cooperative skills are represented as Probabilistic Interaction Primitives which can capture variance and inherent correlations in the demonstrations. We evaluate our method with different human subjects in a task where a robot assists the subjects in making a salad. We also evaluate hereby how learned skills transfer between different subjects. Second, intelligent assistive robots should be able to adapt learned skills to humans when working in close contact or shared workspaces. For Probabilistic Movement Primitives (ProMPs), which were chosen as a skill representation in this thesis, such methods for online adaptation were missing in the literature so far. Hereby, it is in particular important to also evaluate the perceived level of safety and comfort of humans according to different adaptation strategies. To this end, we present two methods for online adaptation of learned skills in a shared workspace setting. Here, we introduce two novel online adaptation methods for ProMPs, namely spatial deformation and temporal scaling. Spatial deformation avoids collisions by dynamically changing the shape of the movement primitive, while at the same time staying close to the demonstrated motions. In temporal scaling, we adapt the ProMP's velocity profile to avoid time-dependent collisions. To achieve intention aware adaptation in shared workspaces we combine both methods with a goal-directed prediction model for human motions. This prediction model can also be learned online from human motions. We conducted experiments for both novel adaptation methods in comparison to non-adaptive behavior with inexperienced users and evaluated influences on task performance as well as subjective metrics such as comfort and perceived level of safety. The third challenge that we consider in this thesis is how a library of learned skills can be used in practice to solve sequential robotic tasks. While hereby reinforcement learning offers a powerful tool for reward-driven learning and self-improvement, in real robotic applications it often suffers from costly and time-consuming sample collection. Here, human input might be beneficial to speed up and guide the learning. Therefore, it is important to enable and compare different ways how human input can be incorporated in reinforcement learning algorithms. In this thesis, we present an approach, which incorporates multiple forms of human input into reinforcement learning for sequential tasks. Since depending on the task human input might not always be correct, we additionally introduce the concept of self-confidence for the robot, such that it becomes able to question human input. We evaluate which input channels humans prefer during interaction and how well they accept suggestions or rejections of the robot if the robot becomes confident in its own decisions. To summarize, the different parts of the thesis contribute to the development of intelligent assistive robots that can learn from imitating humans, adapt the learned skills dynamically to humans in shared workspaces and profit and learn from human input during self-driven learning of how to sequence skills into more complex tasks. The three main contributions to the state of the art are hereby: First, a novel approach to incrementally learn a library for collaborative skills when the total number of skills is not known a priori. Second, two novel methods for online adaptation of ProMPs and their combination with a goal-directed prediction model to enable intention aware online adaptation in shared workspaces. And third, an approach that combines multiple forms of human input with a reinforcement learning algorithm and a novel concept of self-confidence to learn and improve the sequencing of skills into more complex tasks