Multiclass recognition and part localization with humans in the loop

We propose a visual recognition system that is designed for fine-grained visual categorization. The system is com-posed of a machine and a human user. The user, who is un-able to carry out the recognition task by himself, is interac-tively asked to provide two heterogeneous forms of informa-tion: clicking on object parts and answering binary ques-tions. The machine intelligently selects the most informative question to pose to the user in order to identify the object’s class as quickly as possible. By leveraging computer vision and analyzing the user responses, the overall amount of hu-man effort required, measured in seconds, is minimized. We demonstrate promising results on a challenging dataset of uncropped images, achieving a significant average reduc-tion in human effort over previous methods. 1