language:mk license:cc-by-sa-3.0 size_categories:10K<n<100K format:json modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: cc-by-sa-3.0
language:
- mk
---

# NQ-Open MK version

This dataset is a Macedonian adaptation of the [NQ-Open dataset](https://huggingface.co/datasets/gordicaleksa/serbian-llm-eval-v1), originally curated (English -> Serbian) by Aleksa Gordić. It was translated from Serbian to Macedonian using the [Google Translate API](https://cloud.google.com/translate).

You can find this dataset as part of the macedonian-llm-eval [GitHub](https://github.com/LVSTCK/macedonian-llm-eval) and [HuggingFace](https://huggingface.co/datasets/LVSTCK/macedonian-llm-eval).

NOTE: train version of the dataset is not fully complete, as there are about 66k instances instead of 87k (Google Translation API budget exceeded :-) ) 

## Why Translate from Serbian?
The Serbian dataset was selected as the source instead of English because Serbian and Macedonian are closer from a linguistic standpoint, making Serbian a better starting point for translation. Additionally, the Serbian dataset was refined using GPT-4, which, according to the original report, significantly improved the quality of the translation. Note that this is an assumption that needs further validation (quantitative).. a small quality check was conducted on the translated Macedonian dataset, and the results were deemed to be of good quality. 

## Overview
- **Language**: Macedonian
- **Source**: Serbian LLM Evaluation Dataset v1
- **Translation**: Serbian -> Macedonian (via Google Translate API)
- **Split**: Train (partial) and test split are available.
- **Purpose**: Benchmark and evaluate large language models in Macedonian.

## Credits
- Serbian version of the dataset: [Aleksa Gordić](https://huggingface.co/datasets/gordicaleksa/serbian-llm-eval-v1)
- Translation: [Google Translate API](https://cloud.google.com/translate)
- Original dataset - NQ-Open Paper: [Natural Questions: A Benchmark for Question Answering Research](https://aclanthology.org/Q19-1026/)

## Limitations
- Translation may contain minor inaccuracies.

## Citation 
If you use this dataset, please cite the original NQ-Open dataset:

```
@article{doi:10.1162/tacl\_a\_00276,
    author = {Kwiatkowski, Tom and Palomaki, Jennimaria and Redfield, Olivia and Collins, Michael and Parikh, Ankur and Alberti, Chris and Epstein, Danielle and Polosukhin, Illia and Devlin, Jacob and Lee, Kenton and Toutanova, Kristina and Jones, Llion and Kelcey, Matthew and Chang, Ming-Wei and Dai, Andrew                         M. and Uszkoreit, Jakob and Le, Quoc and Petrov, Slav},
    title = {Natural Questions: A Benchmark for Question Answering Research},
    journal = {Transactions of the Association for Computational Linguistics},
    volume = {7},
    number = {},
    pages = {453-466},
    year = {2019},
    doi = {10.1162/tacl\_a\_00276},
    URL = { 
            https://doi.org/10.1162/tacl_a_00276
        },
    eprint = { 
            https://doi.org/10.1162/tacl_a_00276
        
        },
    abstract = { We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature. }
}

@inproceedings{lee-etal-2019-latent,
    title = "Latent Retrieval for Weakly Supervised Open Domain Question Answering",
    author = "Lee, Kenton  and
      Chang, Ming-Wei  and
      Toutanova, Kristina",
    booktitle = "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2019",
    address = "Florence, Italy",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/P19-1612",
    doi = "10.18653/v1/P19-1612",
    pages = "6086--6096",
    abstract = "Recent work on open domain question answering (QA) assumes strong supervision of the supporting evidence and/or assumes a blackbox information retrieval (IR) system to retrieve evidence candidates. We argue that both are suboptimal, since gold evidence is not always available, and QA is fundamentally different from IR. We show for the first time that it is possible to jointly learn the retriever and reader from question-answer string pairs and without any IR system. In this setting, evidence retrieval from all of Wikipedia is treated as a latent variable. Since this is impractical to learn from scratch, we pre-train the retriever with an Inverse Cloze Task. We evaluate on open versions of five QA datasets. On datasets where the questioner already knows the answer, a traditional IR system such as BM25 is sufficient. On datasets where a user is genuinely seeking an answer, we show that learned retrieval is crucial, outperforming BM25 by up to 19 points in exact match.",
}
```