task_categories:text-generation task_categories:automatic-speech-recognition language:ar language:en license:cc-by-sa-4.0 size_categories:100K<n<1M format:text modality:text library:datasets library:mlcroissant region:us
---
license: cc-by-sa-4.0
task_categories:
- text-generation
- automatic-speech-recognition
language:
- ar
- en
---
# ArE-CSTD: Arabic-English Code-Switching Textual Dataset

The National Center for Artificial Intelligence at the Saudi Data and Artificial Intelligence Authority (SDAIA), published the "ArE-CSTD" dataset, which stands for "Arabic-English Code-Switching Textual Dataset”.

This dataset contains 330K dialectical Arabic-English code-swithing sentences generated by the large language model GPT-4.

## TXT Files

There are 6 txt files. 2 files for Modern Standard Arabic(MSA) train and test splits, 2 for Saudi Arabic dialect train and test splits, and 2 for Egyptian Arabic dialect train and test splits. In each file, each new line has a new sentence.

**Note**: Processing includes removing duplicated sentences (exact match), removing sentences containing Arabizi words or expressions, and inserting spaces between Arabic and English words. 

## Datasets Statistics
The following table shows a detailed statistics.

| Statistics  | Value |
| --- | :---: |
| **Total # of Sentences** | 330K |
| **Total # of Tokens** | 3M |
| **Total # of Arabic Tokens** | 1.6M |
| **Total # of English Tokens** | 1.3M |
| **Ratio of English to Arabic Tokens** | 1:1.25 |
| **Avg. Words per Sentence** | 9.3 |
| **Avg. English Words per Sentence** | 4.1 |
| **Avg. CMI of all Sentence** | 14% |
| **# of MSA Tokens** | 1.2M |
| **# of SA Tokens** | 1.2M |
| **# of EGY Tokens** | 1.2M |

## Licenses 

This work is licensed under a CC BY-NC-SA 4.0 license.

## Citation

If you use ArE-CSTD dataset please use the following citation:

```
@inproceedings{ArECSTD2024,
  Title = {Leveraging LLM for Augmenting Textual Data in Code-Switching ASR: Arabic as an Example},
  Author = {Sadeen Alharbi, Raghad Aloraini, Reem BinMuqbil, Ahmed Ali, Saiful Bari, Areeb Alowisheq, Yaser Alonaizan},
  Booktitle = {To be published},
  affiliation = {NCAI-SDAIA}
  Year = {2024}
}
```
________________________________

File last update: JUL 28, 2024.