license:apache-2.0 size_categories:10K<n<100K format:csv modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: apache-2.0
---

# Q&A Quantum Information Dataset

This dataset is created by digesting `500` different papers from the `quantum information` directory on `arXiv`, 
the papers are located based on their relevance to the `quantum information` keyword.

## Data retrieval

The data is extracted from the `.pdf` files using `PyMuPDF` package proxied from `langchain`.

Then the Q&A pair is generated by:
1. Generate `N` questions per page of the `.pdf` document based on its content.
2. We will feed each question to an LLM backed with a vector store generated from the article content, and get the answer.

## Data distillation

Since the quality of the generated questions is undetermined, we will run a distillation process on the Q&A pair:
1. Ask an LLM to rate the question according to its relevance to a set of statements.
2. Ask an LLM to rate the completeness of the question (i.e. unexplained reference to background context, undefined abbreviations)
3. Filter the Q&A pair based on a threshold for the ratings.
