task_categories:question-answering task_categories:text-classification task_categories:text-generation language:en license:apache-2.0 size_categories:100K<n<1M format:json modality:text library:datasets library:pandas library:mlcroissant library:polars arxiv:2408.04957 region:us
---
license: apache-2.0
task_categories:
- question-answering
- text-classification
- text-generation
language:
- en
pretty_name: LLaVA-VSD-120K
size_categories:
- 100K<n<1M
---
# LLaVA-VSD-120K
120K Visual Spatial Description dataset for instruction-tuning Large Language-and-Vision Assistant.

## Dataset Details
Visual Spatial Description (VSD) aims to generate texts that describe the spatial relationships between objects within images.
Traditional visual spatial relationship classification (VSRC) methods typically output the spatial relationship between two objects in an image, 
often neglecting world knowledge and lacking general language capabilities. 
We propose a Large Language-and-Vision Assistant for Visual Spatial Description, named LLaVA-VSD, which is designed for the classification, description, and open-ended description of visual spatial relationships.
Specifically, we first construct a VSD instruction-following dataset using given figure-caption pairs for the three tasks.
It then employs LoRA to fine-tune a Multimodal Large Language Model for VSD. 



### Dataset Sources

<!-- Provide the basic links for the dataset. -->

**[arxiv](https://www.arxiv.org/pdf/2408.04957)**
**[github](https://github.com/swordlidev/LLaVA-VSD)**


## Uses

You can use this VSD dataset to fine-tune a Multimodal Large Language Model for Visual Spatial Description


## Dataset Structure

- `LLaVA-VSD-120K`
  - `images`
  - `label`                   
  - `language`               
    - `llava_vsd_train.json`
    - `llava_vsd_val_task1.json`
    - `llava_vsd_val_task2.json`
    - `llava_vsd_val_task3.json`





### Who are the source data producers?

https://github.com/LLLogen/VSDcode


**BibTeX:**

```
@article{jin2024llava,
  title={LLaVA-VSD: Large Language-and-Vision Assistant for Visual Spatial Description},
  author={Jin, Yizhang and Li, Jian and Zhang, Jiangning and Hu, Jianlong and Gan, Zhenye and Tan, Xin and Liu, Yong and Wang, Yabiao and Wang, Chengjie and Ma, Lizhuang},
  journal={arXiv preprint arXiv:2408.04957},
  year={2024}
}
```

## Dataset Card Contact

swordli@tencent.com