task_categories:text-retrieval task_categories:image-to-text task_categories:visual-question-answering language:en license:apache-2.0 size_categories:1M<n<10M format:parquet modality:text library:datasets library:dask library:mlcroissant library:polars arxiv:2411.08334 region:us
---
license: apache-2.0
task_categories:
- text-retrieval
- image-to-text
- visual-question-answering
language:
- en
size_categories:
- 1M<n<10M
dataset_info:
  features:
  - name: image
    dtype: string
  - name: id
    dtype: string
  - name: question
    dtype: string
  - name: document
    dtype: string
  - name: answer
    dtype: string
  - name: source
    dtype: string
  splits:
  - name: train
    num_bytes: 1862423832
    num_examples: 1334583
  download_size: 996324430
  dataset_size: 1862423832
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
---

# MIRe Pre-training Dataset for Multimodal Query Retrieval

This repository contains the pre-training dataset used in our work on [**MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval**](https://arxiv.org/abs/2411.08334). The dataset is designed for training multimodal retrieval systems that integrate both visual and textual cues without fusing text features during the alignment stage.

**Note:** This release excludes data from the WiT corpus.

## Overview

The MIRe Pre-training Dataset consists of multimodal query–passage pairs generated by converting concise question–answer pairs into extended passages. Each sample includes an image paired with a textual query and a corresponding extended passage constructed via our response-to-passage conversion process. This design better mimics real-world retrieval tasks by exposing the model to more complex and noisy documents.

Key statistics:
- **Total Samples:** 1,356,536 QA pairs
- **Images:** 264,262 unique images
- **Queries per Image:** On average, 8.32 queries (with up to 12 queries per image)
- **Query Types:** Approximately 17% of queries require detailed descriptions; the remainder cover a variety of query types

The dataset is built from multiple sources such as ST-VQA, TextVQA, LLaVAR, Instruct4V, and LLaVA-1.5, ensuring diverse query-response pairs while excluding WiT data.

## Data Structure

Each entry in the dataset contains the following fields:

- **`image`**: The image file.
- **`question`**: The original textual query associated with the image.
- **`answer`**: The concise answer or response (as originally provided).
- **`document`**: An extended passage generated by merging the response with top-retrieved passages from a knowledge base (via the response-to-passage conversion process).
- **`source`**: The originating dataset (e.g., ST-VQA, TextVQA, etc.).

This structure facilitates experiments in multimodal retrieval by linking visual content with rich textual information.

## Usage

You can easily load the dataset using the Hugging Face Datasets library. For example:

```python
from datasets import load_dataset

dataset = load_dataset("Y-J-Ju/MIRe_ViD2R")
print(dataset)
```

This code will load the dataset splits (e.g., train, validation) and display the fields for each sample.

## Citation

If you use this dataset, please cite our paper:

> Yeong-Joon Ju, Ho-Joong Kim, Seong-Whan Lee. "MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval.".

## License

This dataset is released under the Apache-2.0 license. Please refer to the LICENSE file for further details.

## Additional Information

- **Response-to-Passage Conversion:** The extended passages are generated by augmenting the original responses with context retrieved from a large text corpus. This process bridges the gap between concise responses and the more detailed passages needed for effective retrieval training.
- **Excluding WiT Data:** This release does not include WiT data, ensuring that the dataset only comprises samples from the visual instruction and VQA sources mentioned above.

For further details on the data construction process and experimental setup, please refer to the MIRe paper.