license:mit size_categories:100K<n<1M format:parquet modality:tabular modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: mit
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
dataset_info:
  features:
  - name: query_id
    dtype: int64
  - name: text
    dtype: string
  - name: document_ids
    sequence: string
  - name: scores
    sequence: float64
  - name: means
    dtype: float64
  - name: stds
    dtype: float64
  - name: maxmins
    dtype: float64
  splits:
  - name: train
    num_bytes: 351849640
    num_examples: 500000
  download_size: 198040061
  dataset_size: 351849640
---

This dataset is based on the translated MMARCO dataset. 
500k queries (from [queries](https://huggingface.co/datasets/akhooli/ar_mmarco_queries)) were selected and scored against 32 documents each 
(based on English scores) from 
[Ar MMARCO Docs](https://huggingface.co/datasets/akhooli/ar_mmarco_docs) (translated) using JinaAI multilingual reranker. The stats columns (mean, std and maxmin) are used to 
curate subsets for training. 
The dataset can be used to train knowledge distillation ColBERT models and some are available under the models section.  
Note: translation quality is not good enough and scores are not spread apart. Some statistics were applied to curate a subset with wide distribution but this is not enough.