MIntRec2.0: A Novel Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations

Multimodal intent recognition poses significant challenges, requiring the incorporation of non-verbal modalities from real-world contexts to enhance the comprehension of human intentions. However, most existing multimodal intent benchmark datasets are limited in scale and suffer from handling out-of-scope samples that arise in multi-turn conversational interactions. In this paper, we introduce MIntRec2.0, a novel benchmark dataset for multimodal intent recognition in multi-party conversations. It comprises of 1,245 dialogues collected from three TV series, encompassing 15,040 high-quality samples with text, video, and audio information. We expand the existing intent taxonomy to 30 fine-grained intent classes and annotate over 9,300 in-scope and 5,700 out-of-scope instances. This allows for evaluating the effectiveness of both in-scope intent recognition as well as robustness in detecting out-of-scope samples. Moreover, the dataset provides information about the different speakers involved in each dialogue, enabling both single-turn and multi-turn conversational multimodal intent recognition. To demonstrate the efficacy of utilizing multimodal information in conversational intent recognition, we employ classic multimodal fusion methods as benchmark methods. Furthermore, evaluation benchmarks are built with ChatGPT and humans, revealing a substantial performance gap between large language models and humans. To the best of our knowledge, MIntRec2.0 is the first large-scale multimodal dataset for intent recognition and out-of-scope detection, providing a pioneering foundation for further research in this field. The dataset and codes can be accessed through the links provided in the supplementary materials