Mining Semantic Affordances of Visual Object Categories

Affordances are fundamental attributes of objects. Affor-dances reveal the functionalities of objects and the possible actions that can be performed on them. Understanding af-fordances is crucial for recognizing human activities in vi-sual data and for robots to interact with the world. In this paper we introduce the new problem of mining the knowl-edge of semantic affordance: given an object, determining whether an action can be performed on it. This is equiv-alent to connecting verb nodes and noun nodes in Word-Net, or filling an affordance matrix encoding the plausibility of each action-object pair. We introduce a new benchmark with crowdsourced ground truth affordances on 20 PASCAL VOC object classes and 957 action classes. We explore a number of approaches including text mining, visual mining, and collaborative filtering. Our analyses yield a number of significant insights that reveal the most effective ways of collecting knowledge of semantic affordances. 1