task_categories:text-generation language:zh size_categories:10M<n<100M format:parquet modality:text library:datasets library:dask library:mlcroissant library:polars region:us
---
dataset_info:
  features:
  - name: passage
    dtype: string
  splits:
  - name: train
    num_bytes: 18996999214
    num_examples: 88328203
  download_size: 13088559046
  dataset_size: 18996999214
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
task_categories:
- text-generation
language:
- zh
size_categories:
- 10M<n<100M
---
# Dataset Card for "clean_passages_80m-chinese-zhtw"

包含**8千萬餘萬**（88328203）個中文段落，不包含任何字母、數字。文字長度大部分介於 50\~200 個字。

原始資料集是用於訓練[GENIUS模型中文版](https://huggingface.co/spaces/beyond/genius)。論文參考引用:

```
@article{guo2022genius,
  title={GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation},
  author={Guo, Biyang and Gong, Yeyun and Shen, Yelong and Han, Songqiao and Huang, Hailiang and Duan, Nan and Chen, Weizhu},
  journal={arXiv preprint arXiv:2211.10330},
  year={2022}
}
```

## 資料集來源

本資料集是基於[CLUE中文預訓練語料集](https://github.com/CLUEbenchmark/CLUE)進行處理、過濾并進行簡繁轉諲而得到的。

原始資料集引用:

```
@misc{bright_xu_2019_3402023,
author       = {Bright Xu},
title        = {NLP Chinese Corpus: Large Scale Chinese Corpus for NLP },
month        = sep,
year         = 2019,
doi          = {10.5281/zenodo.3402023},
version      = {1.0},
publisher    = {Zenodo},
url          = {https://doi.org/10.5281/zenodo.3402023}
}
```
