task_categories:text-classification task_categories:question-answering task_categories:zero-shot-classification language:bm language:ses language:snk language:bmq language:bze language:myk license:cc-by-nc-4.0 size_categories:100K<n<1M format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: cc-by-nc-4.0
configs:
- config_name: default
  data_files:
  - split: bam
    path: data/bam-*
  - split: boz
    path: data/boz-*
  - split: bom
    path: data/bom-*
  - split: myk
    path: data/myk-*
  - split: sgh
    path: data/sgh-*
  - split: snk
    path: data/snk-*
dataset_info:
  features:
  - name: original
    dtype: string
  - name: cleaned
    dtype: string
  - name: space_insertion
    dtype: string
  - name: char_delete
    dtype: string
  - name: char_swap
    dtype: string
  - name: word_swap
    dtype: string
  - name: word_delete
    dtype: string
  - name: word_blank
    dtype: string
  splits:
  - name: bam
    num_bytes: 51067075
    num_examples: 100000
  - name: boz
    num_bytes: 352249
    num_examples: 254
  - name: bom
    num_bytes: 3097936
    num_examples: 3085
  - name: myk
    num_bytes: 22544040
    num_examples: 32916
  - name: sgh
    num_bytes: 39044992
    num_examples: 56126
  - name: snk
    num_bytes: 2510973
    num_examples: 8181
  download_size: 85681755
  dataset_size: 118617265
task_categories:
- text-classification
- question-answering
- zero-shot-classification
language:
- bm
- ses
- snk
- bmq
- bze
- myk
pretty_name: mlnoisy
size_categories:
- 100K<n<1M
version: 0.1.1
---

# ml.Datx - Noisy

This dataset contains texts in six Malian languages with various noise functions applied. It is designed for use with Hugging Face's NLP tools and models.

## Languages Included
- Bambara
- Bomu
- Bozo
- Mamama
- Songhay
- Soninke

## Noise Functions Applied
The following noise functions have been applied to the texts:
- **Character Swap**: Randomly swaps adjacent characters.
- **Random Space Insertion**: Inserts random characters at random positions.
- **Random Character Deletion**: Deletes random characters.
- **Random Word Position Swapping**: Substitutes random characters with other random characters.
- **Word Masking**: Random word masking using BLANK keyword

## Dataset Structure
The dataset is structured as follows:
- data/lang_id_output.json`: Contains a list based dictionary representation of the data

Each data instance contains the followings *original*, *space_insertion*, *char_delete*, *char_swap*, *word_swap*, *word_delete*, *word_blank*, *cleaned*, for example:

```json
[
    {
        "original": "Yesu Kirisita Tuura Wema TUURA WEMA",
        "space_insertion": " Ye su Kiris i ta Tu u r a  Wem a  TUU RA   WEM A",
        "char_delete": "Ysu isia Tuurema TURAWMA",
        "char_swap": "eYsu iKisriat Turua Wma eTUURA WMEA",
        "word_swap": "Yesu Kirisita Tuura Wema WEMA TUURA",
        "word_delete": "Yesu Kirisita Tuura Wema TUURA WEMA",
        "word_blank": "Yesu Kirisita Tuura Wema BLANK WEMA",
        "cleaned": "Yesu Kirisita Tuura Wema TUURA WEMA"
    }
...
]
```

## Usage
To use this dataset with Hugging Face, you can load it using the `datasets` library:

```python
from datasets import load_dataset

dataset = load_dataset('mlsftwrs/mlnoisy')
```
## Sources
- TBA

## Citation
If you use this dataset in your research, please cite it as follows:

```
@misc{diarra2025mldatxnoisy,
    title={ml.Datx:Noisy -  Malian Noisy Dataset for LLM Training},
    author={
        Sebastien Diarra and
        Nene Tangara and
        Seydou Diallo
    },
    url={https://mlsftwrs.ml/project/mldatx},
    year={2025}
}
```

## Contact
For any questions or issues, please contact: 
- sdiarra@robotsmali.org.