language:en license:cc-by-sa-4.0 size_categories:1K<n<10K format:csv modality:text modality:video library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: cc-by-sa-4.0
language:
- en
pretty_name: Multimodal Intent Recognition (MIntRec)
---

## Dataset details
In real-world conversational interactions, we usually combine information from multiple modalities (e.g., text, video, audio) to help analyze human intentions. Though intent analysis has been widely explored in the Natural Language Processing community, there is a scarcity of data for multimodal intent analysis. Thus, we provide a novel multimodal intent benchmark dataset, MIntRec, to boom the research. To the best of our knowledge, it is the first multimodal intent dataset from real-world conversational scenarios.

## Dataset Construction:
## a. Data sources
We collect raw data from the Superstore TV series. The reasons are that it contains (1) a wealth of characters (including seven prominent and twenty recurring roles) with different identities in the superstore and (2) a mass of stories in various scenes (e.g., shopping mall, warehouse, office).
## b. Intent taxonomies
In this work, we design new hierarchical intent taxonomies for multimodal scenes. Inspired by human intention philosophy and goal-oriented intentions in artificial intelligence research, we categorize two coarse-grained intent categories: "Express emotions or attitudes" and "Achieve goals".

We further categorize the two coarse-grained intent classes into 20 fine-grained classes by analyzing as many video segments and summarizing high-frequency intent tags. They are as follows:
##   Express emotions and attitudes:
Complain, Praise, Apologize, Thank, Criticize, Care, Agree, Taunt, Flaunt, Oppose, Joke
##   Achieve goals:
Inform, Advise, Arrange, Introduce, Comfort, Leave, Prevent, Greet, Ask for help.

## c. Multimodal Intent Annotation
Five annotators label the full dataset independently. They need to combine text, video, and audio information and determine one intent label with the most confidence. The qualified samples are saved with votes no less than three notes among the twenty fine-grained intent classes.

**Dataset Distribution**
![image/png](https://cdn-uploads.huggingface.co/production/uploads/64b646c00eda977859dcb1af/ZLA9PVr451khlfgRXlqPB.png)

**Dataset Statistics**

![image/png](https://cdn-uploads.huggingface.co/production/uploads/64b646c00eda977859dcb1af/3UdMTPjGnL-TEyyLaiyNP.png)


**Where to send questions or comments about the model:**
https://github.com/thuiar/MIntRec/issues

## Intended use
**Primary intended uses:**
The primary use of MIntRec is to combine information from multiple modalities (e.g., text, video, audio) to help analyze human intent.
