license:mit region:us
---
license: mit
---

# MIRAGE Pretraining/Finetuning Dataset Card

## Dataset details
1. Dataset type: This dataset is designed to train the visual-RAG model, MIRAGE-8.3B. It contains files to do (multi-stage) pre-training as well as fine-tuning.

2. Data Preparation:

- Stage1 Pretraining: Q-Former and visual alignment layer (low-quality data)
    - Source: LAION-400M, CC12M, and MSCOCO from [here](https://github.com/rom1504/img2dataset)
    - Put all these `.tar` files under `/datasets` directory. [stage1_pretraining.txt](./stage1_pretraining.txt) provides an example dataset.
- Stage2 Pretraining: Q-Former and visual alignment layer (high-quality data)
    - Source: [Share-GPT4V Captioning Data](https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/)
    - Please follow their repo to download the image to the right place (in the `playground/data`) accordingly.
- Stage3 Pretraining: Retriever
    - Source: Augmented [LLaVA Instruction Tuning Data](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K)
    - Put all images into (in the `playground/data`)
- Instruction Finetuning
    - Source: Augmented [LLaVA Instruction Tuning Data](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K), [RETVQA](https://vl2g.github.io/projects/retvqa/), [WebQA](https://drive.google.com/drive/folders/1ApfD-RzvJ79b-sLeBx1OaiPNUYauZdAZ), [SlideVQA](https://drive.google.com/file/d/11bsX48cPpzCfPBnYJgSesvT7rWc84LpH/view)

- In brief, the data structure should look like this:
  ```bash
  playground/data/
  ├── coco
  │   ├── annotations
  │   ├── test2017
  │   ├── train2017
  │   └── val2017
  ├── gqa
  │   └── images
  ├── ocr_vqa
  │   └── images
  ├── sam
  │   └── images 
  ├── share_textvqa
  │   └── images
  ├── slidevqa
  │   └── images (download it from https://drive.google.com/file/d/11bsX48cPpzCfPBnYJgSesvT7rWc84LpH/view)
  ├── textvqa
  │   └── train_images
  ├── vg
  │   ├── VG_100K
  │   └── VG_100K_2
  └── webqa
    └── webqa_images (download it from https://drive.google.com/drive/folders/1ApfD-RzvJ79b-sLeBx1OaiPNUYauZdAZ and turn them into xxx.jpg)
  ```

3. Please check out our [project page](https://visual-haystacks.github.io) for more information. You can also send questions or comments about the model to [our github repo](https://github.com/visual-haystacks/mirage/issues).

## Intended use
Primary intended uses: The primary use of VHs is research on large multimodal models and chatbots.

Primary intended users: The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.
