task_categories:text-generation language:zh license:apache-2.0 size_categories:10K<n<100K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars arxiv:2311.17487 region:us
---
dataset_info:
  config_name: '20250216'
  features:
  - name: conversations
    list:
    - name: content
      dtype: string
    - name: role
      dtype: string
  - name: dataset
    dtype: string
  splits:
  - name: train
    num_bytes: 124216883
    num_examples: 25756
  download_size: 69760700
  dataset_size: 124216883
configs:
- config_name: '20250216'
  data_files:
  - split: train
    path: 20250216/train-*
license: apache-2.0
task_categories:
- text-generation
language:
- zh
---
# TWLLM-Data: Open Traditional Mandarin LLM Conversations

TWLLM-Data is the first large-scale open dataset containing real user-generated conversation logs from [TWLLM](https://www.twllm.com) and [TWLLM Arena](http://arena.twllm.com), where over 80% of users are based in Taiwan. The dataset is designed to facilitate the development and evaluation of Traditional Mandarin Large Language Models (LLMs).

## Dataset Overview

- **Source**: User interactions from TWLLM logs.
- **Size**: ~25K conversations.
- **Format**: `parquet` (JSON-like structured dialogues).
- **Languages**: Traditional Mandarin (`zh-TW`).
- **License**: [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0).

### Data Collection & Processing

1. **Real User Prompts**: Extracted from **twllm.com** and **arena.twllm.com** logs, ensuring authentic Taiwanese user input.
2. **Model Response Replacement**: The original model responses were removed and replaced with high-quality answers generated by **Gemini, Claude, or ChatGPT**.
3. **Synthetic Prompting Subsets**:
   - `twllm-evol` and `twllm-synthetic`: Prompts generated using evol-instruct-style transformations.

This dataset is intended for research in **fine-tuning Traditional Mandarin LLMs** and **understanding real-world user queries**.

## Usage

You can load the dataset with:

```python
from datasets import load_dataset

dataset = load_dataset("yentinglin/twllm-data")
```

## Citation

If you use this dataset in your research, please cite:

```
@article{DBLP:journals/corr/abs-2311-17487,
  author       = {Yen{-}Ting Lin and
                  Yun{-}Nung Chen},
  title        = {Taiwan {LLM:} Bridging the Linguistic Divide with a Culturally Aligned
                  Language Model},
  journal      = {CoRR},
  volume       = {abs/2311.17487},
  year         = {2023},
  url          = {https://doi.org/10.48550/arXiv.2311.17487},
  doi          = {10.48550/ARXIV.2311.17487},
  eprinttype    = {arXiv},
  eprint       = {2311.17487},
  timestamp    = {Tue, 05 Dec 2023 14:40:42 +0100},
  biburl       = {https://dblp.org/rec/journals/corr/abs-2311-17487.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}
```

## Contact

For any inquiries or contributions, reach out to [Yenting Lin](mailto:ytl@ieee.org).