language:en license:apache-2.0 size_categories:n<1K format:parquet modality:image modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: apache-2.0
language:
- en
size_categories:
- n<1K
configs:
  - config_name: default
    data_files:
    - split: test
      path: test.parquet
  - config_name: for_image_viewer
    data_files:
      - split: images
        path: images/test-*.parquet
dataset_info:
- config_name: default
  features:
  - name: qid
    dtype: string
  - name: image_files
    sequence: string
  - name: context
    dtype: string
  - name: question
    dtype: string
  - name: answer
    dtype: int64
  splits:
  - name: test
    num_bytes: 50321
    num_examples: 100
  download_size: 29430
  dataset_size: 50321
- config_name: for_image_viewer
  features:
  - name: image
    dtype: image
  - name: exam_url
    dtype: string
  - name: license
    struct:
    - name: url
      dtype: string
    - name: id
      dtype: int64
    - name: name
      dtype: string
  - name: date_captured
    dtype: string
  - name: height
    dtype: int64
  - name: width
    dtype: int64
  - name: id
    dtype: int64
  - name: coco_image_file_name
    dtype: string
  - name: coco_url
    dtype: string
  - name: flickr_url
    dtype: string
  splits:
  - name: images
    num_bytes: 40833267
    num_examples: 285
  download_size: 38578714
  dataset_size: 40833267
---
# Dataset Card for DiscriBench

DiscriBench is an English, multiple-choice, VQA dataset containing 100 samples to evaluate, analyze, and diagnose the discriminative capability of Vision-Language Models (VLMs).

## Dataset Details

<!-- ### Dataset Description -->

- **Language(s):** English
- **License:** Apache-2.0. Images have separete licenses. See [metadata.jsonl](images/metadata.jsonl).

## Uses

### Direct Use

This dataset is expected to be used to evaluate, analyze, and diagnose the discriminative capability of VLMs.

### Out-of-Scope Use

This dataset is not expected to be used for any purpose other than the above.

## Quick start

```python
import json
import os

from huggingface_hub import snapshot_download
from PIL import Image

local_dataset_path = snapshot_download(repo_id="mmst004/discribench_en", repo_type="dataset")
with open(os.path.join(local_dataset_path, "test.jsonl")) as fi:
    dataset = [json.loads(l) for l in fi]

print(json.dumps(dataset[0], indent=2))
# {
#   "qid": "001",
#   "image_files": [
#     "images/001/0.jpg"
#   ],
#   "context": "Situation: Friends are talking about shopping.\nConversation:\nWoman: What did you buy?\nMan: I looked at some jeans and shirts but got these in the end.\nWoman: Nice! Do you like running?\nMan: Not really, but the design looked cool.",
#   "question": "What did the man buy?",
#   "answer": 3
# }

# load image(s)
images = [Image.open(os.path.join(local_dataset_path, img)).convert("RGB") for img in dataset[0]["image_files"]]
print(images)
# [<PIL.Image.Image image mode=RGB size=1280x1280>]
```

## Dataset Structure

```text
- README.md
- LICENSE
- test.jsonl: List of 100 test cases (*)
- images/
    - XXX/*.jpg: image(s) corresponding to question ID, XXX
    - metadata.jsonl: image metadata containing source URLs and individual image licenses
```

(*) Each test case is formatted as follows:

```text
{
qid: question_id[int],
image_files: List[path_to_image_file[str]],
context: situation_and_conversation[str],
question: question[str],
answer: answer[int],
}
```

## Dataset Creation

Please refer to the paper (to appear) for detail.

### Data Collection and Processing

- Exam: Samples with illustrations are collected from the English listening subject in the Common Test for University Admissions (https://www.dnc.ac.jp/kyotsu/) in Japan, and modified with the following procedures.

  **Details**
  
  The situation description given in Japanese was machine-translated into English and then manually verified them.
  The speaker names in conversations found in the transcript file were modified: from "M:" and "W:" to "Man:" and "Woman:", respectively.
  The machine-translated English situation description was next concatenated with the associated conversation and question obtained from the ground-truth transcript data.
  The option images were extracted and converted from PDF into JPEG, and resized to be 1280x1280.

- COCO: Samples with natural images are collected from COCO [\[Lin et al., 2014\]](https://link.springer.com/chapter/10.1007/978-3-319-10602-1_48) using erroneous agreement [\[Tong et al., 2024\]](https://openaccess.thecvf.com/content/CVPR2024/html/Tong_Eyes_Wide_Shut_Exploring_the_Visual_Shortcomings_of_Multimodal_LLMs_CVPR_2024_paper.html).

### Annotation process

We use Claude 3.5 Sonnet (`20240620`) to generate a prototype of the situation description, conversation, question, and answer for COCO samples. We manually verify and modify if necessary the prototype. Please comply with the [Consumer Terms of Service](https://www.anthropic.com/legal/consumer-terms) provided by Anthropic.


### Personal and Sensitive Information

Faces in images are masked to avoid identifying personal information from them.
Texts are manually verified not to have personal or sensitive information.

## Bias, Risks, and Limitations

The language supported in this dataset is English and the images obtained from COCO may implicitly represent subsets of specific countries/regions/areas where the images are taken (e.g., western countries).
These implicit biases might affect the performance of VLMs evaluated on this dataset, and the results should be carefully handled and understood.

## Acknowledgment

These research results were obtained from the commissioned research (No.22501) by National Institute of Information and Communications Technology (NICT), Japan.
The use of the exam data in the English listening subject in the Common Test for University Admissions was granted by the National Center for University Entrance Examinations.

## Citation

TBA

## License

> Copyright 2025 Masayasu MURAOKA. All rights reserved.
>
> Licensed under the Apache License, Version 2.0 (the "License"); \
> you may not use this file except in compliance with the License. \
> You may obtain a copy of the License at
> 
> http://www.apache.org/licenses/LICENSE-2.0
> 
> Unless required by applicable law or agreed to in writing, software \
> distributed under the License is distributed on an "AS IS" BASIS, \
> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. \
> See the License for the specific language governing permissions and \
> limitations under the License.

## Dataset Card Contact

Please contact @mmst004 for any inqueries.