size_categories:1K<n<10K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
dataset_info:
  features:
  - name: sentence
    dtype: string
  - name: result
    dtype: string
  - name: base word
    dtype: string
  - name: __index_level_0__
    dtype: int64
  splits:
  - name: train
    num_bytes: 183368
    num_examples: 2000
  - name: validation
    num_bytes: 46135
    num_examples: 500
  download_size: 65035
  dataset_size: 229503
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  - split: validation
    path: data/validation-*
---

# Dataset Card for bntng-disambiguation-v1

## Dataset Description

- **Repository:** bntng-disambiguation-v1
- **Paper:** Forthcoming
- **Language:** Jawi (Arabic script adaptation for Malay)
- **Task:** Word Sense Disambiguation

### Dataset Summary

This dataset focuses on word sense disambiguation in Jawi script, specifically addressing cases where the same written form can correspond to different words in Malay. The dataset centers around the ambiguous Jawi writing بنتڠ (bntng) and its various interpretations, with 500 contextual examples for each possible reading. This dataset is designed to test Large Language Models' ability to predict the correct romanized form based on context.

### Languages

- Jawi (Arabic script adaptation for Malay)
- Malay (for disambiguation entries)

### Data Instances

The dataset contains 2500 examples (500 for each variant) of _bntng_, which corresponds to the Jawi form بنتڠ, and that can be interpreted as:
- **banteng** (a type of wild ox)
- **banting** (to take up and throw down)
- **bentang** (to spread out)
- **benteng** (a fortification)
- **bintang** (a star)


### Data Fields

- `sentence`: An example sentence in Rumi (romanized Malay) with bntng.
- `result`: The correct, disambiguated form.


### Data Splits

The dataset contains 2500 total examples, with:
- Training set: 2000 examples (400 per variant)
- Validation set: 500 examples (100 per variant)


## Dataset Creation

### Curation Rationale

The dataset was created to address the challenge of word sense disambiguation in Jawi script, where the absence of short vowel markers can lead to multiple valid interpretations of the same written form. 

### Source Data

#### Initial Data Collection and Normalization

The dataset builds upon teaching examples from Dr. Mulaika Hijjas's work in Jawi education, as documented in her blog post ["Teaching Jawi in the Pandemic"](https://teachingthecodex.com/2022/02/23/teaching-jawi-in-the-pandemic/). The examples were expanded and contextualized by the research team.

### Annotations

The dataset was created and annotated by a team at the National University of Singapore, lead by Miguel Escobar Varela, with each example carefully constructed to provide clear contextual clues for the correct reading.

## Considerations for Using the Data

### Social Impact of Dataset

This dataset contributes to the preservation and processing of Jawi script, which has historical and cultural significance in Southeast Asian Islamic traditions, particularly in Malaysia, Indonesia, and Brunei. It supports the development of digital tools for processing historical Jawi documents.

### Discussion of Biases

- The dataset uses constructed modern examples rather than historical texts, which may introduce contemporary biases in language use and context
- The examples may not fully represent the complexity and variety of historical Jawi usage
- The equal distribution of variants (500 each) is artificial and may not reflect real-world frequency

### Other Known Limitations

- The dataset focuses on a single case of ambiguity (بنتڠ)
- The examples are constructed rather than drawn from historical sources
- Context-dependent disambiguation may still be challenging in some cases

## Technical Information

### Pipeline Integration

This dataset is designed to be used as part of a larger Jawi processing pipeline:
1. OCR Model: [mevsg/qwen-for-jawi-v1](https://huggingface.co/mevsg/qwen-for-jawi-v1)
2. Post-OCR Disambiguation: 
[mevsg/bntng-dis-v1](https://huggingface.co/mevsg/bntng-dis-v1)

## Additional Information

### Dataset Curators

This dataset was created by researchers at the National University of Singapore.

### Licensing Information

This dataset is released under CC BY-SA 4.0 (Creative Commons Attribution-ShareAlike 4.0 International License).

### Citation Information

```bibtex
@misc{bntng2024,
  title={BNTNG-Disambiguation-v1: A Dataset for Jawi Script Word Sense Disambiguation},
  author={Miguel Escobar Varela},
  year={2024},
  howpublished={\url{https://huggingface.co/datasets/mevsg/bntng-disambiguation-v1}},
}
```

### Contributions

Thanks to:
- Dr. Mulaika Hijjas for the initial teaching examples
- The National University of Singapore research team for dataset creation and annotation
