A general framework for high-resolution robotic tactile sensing: design, simulation, and learning

In order to fulfill their potential in the manufacturing and retail sectors of the modern world, autonomous machines need to be able to perceive and react to contact with their surroundings, both to enhance their capabilities, as well as to increase operational safety. This thesis investigates solutions to the contact sensing problem of robotic systems, pivoting on the development of a vision-based tactile sensing principle that provides rich information upon physical interaction with the environment. The sensors based on such a principle are low-cost, scalable to large surfaces and straightforward to manufacture. However, they do not directly measure physical quantities, but rather provide raw data in the form of what are generally known as tactile images. In this work, a machine learning-based data processing framework is presented to address three main requirements, namely, sensing accuracy, efficiency, and generalization across tasks and contact conditions. State-of-the-art sensing accuracy, at a spatial resolution comparable to that of the human fingertip, is achieved through a deep neural network that maps the raw tactile images to the three-dimensional force distribution applied to the sensing surface, which provides a compact and generic representation of the contact state. In fact, the force distribution contains information about the location and the intensity of shear and pressure forces, as well as about the shape and the number of the possibly distinct contact regions. In addition, it provides an interpretable physical quantity that is shown to be very practical for planning higher-level robotic tasks. The size of the neural network is kept compact to ensure real-time inference. However, in the context of data-driven methods, efficiency is also a concern with regard to training data requirements. In this thesis, accurate finite element-based simulations enable the synthetic generation of raw tactile data under a variety of contact conditions. The same simulations also yield appropriate force distribution labels, which are otherwise not possible to collect with currently existing commercial force sensors. Hence, the deep neural network is entirely trained with synthetic data, avoiding the need for real-world data collection. A strategy is then presented that facilitates a seamless transfer of the inference model from simulation to reality, retaining high sensing accuracy. In addition, the model transfers across sensors of the same type without further training. The simulation training facilitates data collection across different scenarios, such as the contact with arbitrarily shaped objects or the combination of shear and pressure interactions. An appropriate choice of learning architecture shows generalization capabilities when applied to contact conditions not present in the training dataset. Beyond the pure sensing task, a proof-of-concept robotic system is presented that fully leverages the versatility of the tactile sensor. The system achieves dynamic manipulation of objects with unknown physical properties solely through the use of tactile feedback fed to a closed-loop control policy trained with a deep reinforcement learning algorithm. In a separate part, this thesis discusses a different research topic, where past experience data are employed to improve the trajectory tracking performance of autonomous systems. This is achieved by estimating unmodeled disturbances over different trials, and including them in the formulation of a computationally efficient model predictive control framework. The approach is demonstrated on two flying vehicle applications, namely, on a vehicle powered with electric ducted fans and controlled through thrust vectoring, and on a quadcopter that aims to balance a pendulum rod during flight