license:gpl-3.0 size_categories:10K<n<100K format:imagefolder modality:image library:datasets library:mlcroissant region:us
---
license: gpl-3.0
---

# VCog-Bench: Benchmarking Multimodal LLMs on Abstract Visual Reasoning

## Description:    

VCog-Bench is a publicly available zero-shot abstract visual reasoning (AVR) benchmark designed to evaluate Multimodal Large Language Models (MLLMs). This benchmark integrates two well-known AVR datasets from the AI community and includes a newly proposed MaRs-VQA dataset. The findings in VCog-Bench show that current state-of-the-art MLLMs and Vision-Language Models (VLMs), such as GPT-4o and LLaVA-1.6, InternVL demonstrate some basic understanding of AVR tasks. However, these models still face challenges with complex matrix reasoning tasks. This highlights the need for further exploration and development in this area. By providing a robust benchmark, we aim to encourage further innovation and progress in the field of zero-shot abstract visual reasoning.

## Benchmark Dataset Structure:     

```
----vcog-bench
    |----cvr
    |    |----case_name1
    |    |    |----answer
    |    |    |    |----image
    |    |    |    |    |----x.png
    |    |    |----choice
    |    |    |    |----image
    |    |    |    |    |----sub_image_0.png
    |    |    |    |    |----sub_image_1.png
    |    |    |    |    |----sub_image_2.png
    |    |    |    |    |----sub_image_3.png
    |    |----case_name2
    |    |----case_name3
    |    |----case_name4
    |    |----......
    |----raven
    |    |----case_name1
    |    |    |----answer
    |    |    |    |----image
    |    |    |    |    |----x.jpeg
    |    |    |----choice
    |    |    |    |----image
    |    |    |    |    |----0.jpeg
    |    |    |    |    |----1.jpeg
    |    |    |    |    |----2.jpeg
    |    |    |    |    |----3.jpeg
    |    |    |    |    |----4.jpeg
    |    |    |    |    |----5.jpeg
    |    |    |    |    |----6.jpeg
    |    |    |    |    |----7.jpeg
    |    |    |    |----text
    |    |    |    |    |----annotation.json
    |    |    |----question
    |    |    |    |----image
    |    |    |    |    |----question.jpeg
    |    |----case_name2
    |    |----case_name3
    |    |----case_name4
    |    |----......
    |----marsvqa
    |    |----case_name1
    |    |    |----answer
    |    |    |    |----image
    |    |    |    |    |----xxx.jpeg
    |    |    |----choice
    |    |    |    |----image
    |    |    |    |    |----xxx.jpeg
    |    |    |    |    |----xxx.jpeg
    |    |    |    |    |----xxx.jpeg
    |    |    |    |    |----xxx.jpeg
    |    |    |    |----text
    |    |    |    |    |----annotation.json
    |    |    |----choiceX
    |    |    |    |----image
    |    |    |    |    |----xxx.jpeg
    |    |    |    |    |----xxx.jpeg
    |    |    |    |    |----xxx.jpeg
    |    |    |    |    |----xxx.jpeg
    |    |    |----question
    |    |    |    |----image
    |    |    |    |    |----xxx.jpeg
    |    |----case_name2
    |    |----case_name3
    |    |----case_name4
    |    |----......
```

## Dataset Details    

Content Types: VQA pairs with multiple images input    
Volume: 560 VQA pairs (RAVEN), 480 VQA pairs (MaRs-VQA), 309 VQA pairs (CVR)      
Source of Data: RAVEN dataset, MaRs-IB, CVR dataset      
Data Collection Method: See the paper      



