using a laptop Training through Observations Action Map Predictions

1 reading a book using a laptop watching TV reading a book Figure 1: We predict regions in 3D scenes where actions are likely to take place. We start by scanning the geometry of real environments using RGB-D sensors and reconstructing a dense 3D mesh (left). We then observe and track people as they interact with the captured environments (mid-left). We use these observations to train a classifier which allows us to infer the likelihood of actions occurring in regions of new, unobserved scenes. We call these predictions action maps and we demonstrate that we are able to deduce action maps for previously unobserved real and virtual scenes (see mid-right and right, respectively). With modern computer graphics, we can generate enormous amounts of 3D scene data. It is now possible to capture high-quality 3D representations of large real-world environments. Large shape and scene databases, such as the Trimble 3D Warehouse, are publicly accessible and constantly growing. Unfortunately, while a great amount of 3D content exists, most of it is detached from the semantics and functionality of the objects it represents. In this paper, we present a method to establish a correlation between the geometry and the functionality of 3D environments. Using RGB-D sensors, we capture dense 3D reconstructions of real-world scenes, and observe and track people as they interact with the environment. With these observations, we train a classifier which can transfer interaction knowledge to unobserved 3D scenes. We predict a like-lihood of a given action taking place over all locations in a 3D en-vironment and refer to this representation as an action map over the scene. We demonstrate prediction of action maps in both 3D scans and virtual scenes. We evaluate our predictions against ground truth annotations by people, and present an approach for characterizing 3D scenes by functional similarity using action maps