task_categories:fill-mask task_categories:text-generation task_ids:language-modeling task_ids:masked-language-modeling annotations_creators:no-annotation language_creators:crowdsourced multilinguality:multilingual source_datasets:original language:en license:cc-by-nc-4.0 size_categories:10M<n<100M region:us text-search
---
annotations_creators:
- no-annotation
language_creators:
- crowdsourced
language:
- en
license: cc-by-nc-4.0
multilinguality:
- multilingual
size_categories:
- 10M<n<100M
source_datasets:
- original
task_categories:
- fill-mask
- text-generation
task_ids:
- language-modeling
- masked-language-modeling
pretty_name: Wiki-DPR
tags:
- text-search
---

The project contains indexes, datasets, checkpoints for RAG training and research.

## Sources
- checkpoint: ColBERTv2 (https://downloads.cs.stanford.edu/nlp/data/colbert/colbertv2/colbertv2.0.tar.gz)
- dataset: wiki_dpr (https://github.com/facebookresearch/DPR/blob/main/dpr/data/download_data.py)
- indexes (https://github.com/ittia-research/check/tree/main/datasets/wiki_dpr)
  - indexing config: `ColBERTConfig(nbits=2, doc_maxlen=220)`
  - Didn't compress the index files to one archive because compress save less than 5% space and takes a long time.

