task_categories:question-answering language:hi license:apache-2.0 size_categories:100K<n<1M format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars arxiv:2404.19254 region:us
---
license: apache-2.0
task_categories:
- question-answering
language:
- hi
size_categories:
- 100K<n<1M
---

# Data Description
This dataset consists of over 100k question answers in Hindi, with 1200 tokens per question on average. Questions are generated from Wikipedia pages (Page title and Chunks). The generated part of data contain Secret Context, Question, Choices, Answer, and Description.

The question will be accompanied with 4 Choices and one and only one of them would be the correct answer. For improving generation quality, a retrieval step is added to extract a chunk of text relevant to the question before generating the tokens of question itself (Secret_Context column). For improving consistency, a description is also asked to be in the response(description column).

# Sample Wiki chunks and their generated outputs:

![image/png](https://cdn-uploads.huggingface.co/production/uploads/640abac896aae649741cd51e/Yyf1Q4qo64Od85p43FuGF.png)

# End to End Workflow:

![image/png](https://cdn-uploads.huggingface.co/production/uploads/640abac896aae649741cd51e/rIGC5jPOEhSvvXB0deAdB.png)

Workflow used for generation is given above and it can be broken down into following steps:

## Creating prompts with context
In this step, wiki dumps are used to create prompts that include context for the question to be answered. The wikidumps for Hindi are preprocessed into json files with page title and page content.

The extracted page contents will be chunked for creating multiple question from the same page. We will be using to creates a chunk with similar text. The chunks that do not meet a specific criteria like length, are filtered. Finally, One-Shot Prompts are created which includes the chunks as context. 
## Creating a dataset from prompts using LLM
In this step, prompts are used by the LLM to generate a response. Then, the generated question and answer pairs are validated to remove cases where the generated response might be wrong. Validation is done using the LLM itself with following questions:
1. Context Relevance: Does the context contain enough information to answer the question?
2. Question Relevance: Is the question related to the context?
3. Answer Accuracy: Is the correct answer accurately marked based on the context provided?
4. Question Clarity: The question should not lead to multiple interpretations. Is the question clear and unambiguous?



Please view https://arxiv.org/abs/2404.19254 for more information