Finding actors and actions in movies

We address the problem of learning a joint model of actors and actions in movies using weak supervision pro-vided by scripts. Specifically, we extract actor/action pairs from the script and use them as constraints in a discrimi-native clustering framework. The corresponding optimiza-tion problem is formulated as a quadratic program under linear constraints. People in video are represented by au-tomatically extracted and tracked faces together with cor-responding motion features. First, we apply the proposed framework to the task of learning names of characters in the movie and demonstrate significant improvements over previous methods used for this task. Second, we explore the joint actor/action constraint and show its advantage for weakly supervised action learning. We validate our method in the challenging setting of localizing and recog-nizing characters and their actions in feature length movies Casablanca and American Beauty. 1