size_categories:n<1K format:parquet modality:image modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
dataset_info:
  features:
  - name: document_filename
    dtype: string
  - name: document_url
    dtype: string
  - name: search_query
    dtype: string
  - name: search_topic
    dtype: string
  - name: search_subtopic
    dtype: string
  - name: search_language
    dtype: string
  - name: search_filetype
    dtype: string
  - name: page_number
    dtype: int32
  - name: page_description
    dtype: string
  - name: page_language
    dtype: string
  - name: page_contains_table
    dtype: bool
  - name: page_contains_figure
    dtype: bool
  - name: page_contains_paragraph
    dtype: bool
  - name: page_image
    struct:
    - name: bytes
      dtype: binary
    - name: path
      dtype: string
  - name: query_type
    dtype: int64
  - name: query_answerability
    dtype: int64
  - name: query_modality
    dtype: int64
  - name: query_language
    dtype: string
  - name: query_reasoning
    dtype: string
  - name: query
    dtype: string
  - name: query_is_self_contained
    dtype: bool
  - name: query_is_self_contained_reasoning
    dtype: string
  - name: answer
    dtype: string
  - name: image_filename
    dtype: string
  - name: __index_level_0__
    dtype: int64
  - name: image
    dtype: image
  splits:
  - name: test
    num_bytes: 27172861.0
    num_examples: 100
  download_size: 26222889
  dataset_size: 27172861.0
configs:
- config_name: default
  data_files:
  - split: test
    path: data/test-*
---

**Filtered dataset for testing from [VDSID](vidore/vdsid)** 

# VDSID: Visual Document Semantic Indexing Dataset

## Overview

The **Visual Document Semantic Indexing Dataset** (VDSID) is a comprehensive resource for information retrieval tasks involving visual documents. It comprises images of document pages alongside associated queries, metadata, and annotations. The dataset is ideal for research in document understanding, semantic indexing, and multimodal information retrieval.

VDSID includes 30,000 queries across 8,000 document pages, extracted from 4,700 documents that cover a wide range of topics and modalities, emphasizing diversity and quality.

## Data Fields

### Document Information

- `document_filename`: Filename of the document.
- `document_url`: Original URL of the document.
- `search_query`: The query used to fetch the document.
- `search_topic`: Topic related to the document.
- `search_subtopic`: Subtopic related to the document.
- `search_language`: Language specified for the search.
- `search_filetype`: Filetype filter applied during the search.

### Page Details

- `page_number`: The page's number within the document.
- `page_description`: A natural language description of the page.
- `page_language`: Language used on the page.
- `page_contains_table`: Boolean indicating the presence of tables.
- `page_contains_figure`: Boolean indicating the presence of figures.
- `page_contains_paragraph`: Boolean indicating the presence of paragraphs.
- `page_image`: Image of the page.

### Query Information

- `query_type`: Type of query (see below).
- `query_answerability`: Answerability level of the query (see below).
- `query_modality`: Modality used for query generation.
- `query_language`: Language of the query.
- `query_reasoning`: Reasoning traces used in query generation.
- `query`: The actual query text.
- `query_is_self_contained`: Boolean indicating if the query is self-contained.
- `query_is_self_contained_reasoning`: Reasoning traces for determining self-contained nature.
- `answer`: Expected answer.

## Query typology

Different question types and answerability levels were designed to distill fine-grained capabilities in retrieval and question-answering models.

### Question Types

- **Extractive:** A clear and specific question that can be answered using only a specific piece of information.
- **Open-ended:** A question that focuses on broad in scope, qualitative aspects of an information.
- **Boolean:** A yes/no question that may involve multiple steps of reasoning.
- **Compare-contrast:** A question that requires comparing and/or contrasting two entities or topics that are closely related to each other.
- **Enumerative:** A question that asks to list all examples that possess a common specific property, optionally requesting details about the specifics of each example.
- **Numerical:** A question about a specific piece of information that can be calculated using data from the page. The question should require more than simply reading numbers directly from the page.

### Answerability Levels

Each generated question has one of the three following answerability levels

- **Fully answerable:** A question is said to be _fully answerable_ if the page contains a precise and complete answer to the question.
- **Partially answerable:** A question is said to be _partially answerable_ if the page contains relevant information that is directly related to the question, but some key information is missing and must be retrieved in other pages or documents in order to give a precise and complete answer.
- **Unanswerable:** A question is said to be _unanswerable_ if the page contains information related to the question's topic or domain but upon closer inspection does not contain information that is useful to answer the question. Those questions are tricky and are meant to test if the retrieval system and/or QA system is able to correctly filter the page when faced with such questions.

## Dataset Creation

The dataset was generated through a multi-step process involving web crawling, document processing, and synthetic query generation.

### Web Crawling

1. **Topic Generation:** User-defined domains and document types were broken down into topics and subtopics using a language model.
2. **Search Query Generation:** Diverse search queries were generated for each subtopic.
3. **Document Retrieval:** Queries were submitted to SerpAPI, applying filters like filetype (PDF and PowerPoint), and the documents found on the first Google Search pages were downloaded.

### Document Processing

Using a three-step pipeline involving Claude 3 Sonnet:

1. **Page Tagging:** Metadata such as captions and the presence of paragraphs, tables, and figures were generated for each page.
2. **Question-Answer Generation:** Question-answer pairs were generated from each document image, with diverse question types and answerability levels.
3. **Self-sufficiency check:** Context-dependent questions are tagged to ensure suitability in retrieval contexts.
