Combining cognitive vision, knowledge-level planning with sensing, and execution monitoring for effective robot control

We describe an approach to robot control in real-world en-vironments that integrates a cognitive vision system with a knowledge-level planner and plan execution monitor. Our approach makes use of a formalism called an Object-Action Complex (OAC) to overcome some of the representational differences that arise between the low-level control mecha-nisms and high-level reasoning components of the system. We are particularly interested in using OACs as a formalism that enables us to induce certain aspects of the representation, suitable for planning, through the robot’s interaction with the world. Although this work is at a preliminary stage, we have implemented our ideas in a framework that supports object discovery, planning with sensing, action execution, and fail-ure recovery, with the long term goal of designing a system that can be transferred to other robot platforms and planners