license:cc0-1.0 size_categories:100K<n<1M format:parquet modality:tabular modality:text library:datasets library:dask library:mlcroissant library:polars region:us
---
license: cc0-1.0
configs:
  - config_name: ell_Grek
    data_files:
    - split: train
      path: ell_Grek*/train-*
  - config_name: eng_Latn
    data_files:
    - split: train
      path: eng_Latn/train-*
  - config_name: heb_Hebr
    data_files:
    - split: train
      path: heb_Hebr/train-*
  - config_name: kor_Hang
    data_files:
    - split: train
      path: kor_Hang/train-*
  - config_name: tur_Latn
    data_files:
    - split: train
      path: tur_Latn/train-*
  - config_name: vie_Latn
    data_files:
    - split: train
      path: vie_Latn/train-*
  - config_name: zho_Hans
    data_files:
    - split: train
      path: zho_Hans/train-*
dataset_info:
  features:
  - name: f
    dtype: string
  - name: o
    dtype: int64
  - name: s
    dtype: int64
  - name: rs
    dtype: int64
  - name: u
    dtype: string
  - name: c
    dtype: string
  - name: ts
    dtype: timestamp[us]
  - name: collection
    dtype: string
  - name: lang
    sequence: string
  - name: prob
    sequence: float64
  - name: text
    dtype: string
  - name: seg_langs
    sequence: string
  - name: robotstxt
    dtype: string
  - name: id
    dtype: string
  - name: filter
    dtype: string
  - name: pii
    sequence:
      sequence: int64
  - name: doc_scores
    sequence: float64
  splits:
  - name: train
    num_bytes: 553035016
    num_examples: 105472
  download_size: 241590644
  dataset_size: 553035016
---

| Language | HPLT-ID | Number of Documents | Number of Subwords |
|----------|---------|---------------------|--------------------|
| Chinese (Simplified)    |zho_Hans | 52,224              |100,409,004         |
| English  |eng_Latn | 114,688             | 100,152,640        |
| Greek    |ell_Grek | 72,192              |100,338,420         |
| Hebrew   |heb_Hebr | 118,784             | 100,014,504        |
| Korean   |kor_Hang | 78,336              | 100,519,317        |
| Turkish  |tur_Latn | 106,496             | 100,132,032        |
| Vietnamese|vie_Latn|105,472              |100,385,036         |

## Data Files Used

Subset directories and data files selected from HPLT/HPLT2.0_cleaned to create subsets.

### Chinese (Simplified)

```
Selected directory: zho_Hans_28
Selected data files:
['zho_Hans_28/train-00076-of-00188.parquet'
 'zho_Hans_28/train-00029-of-00188.parquet'
 'zho_Hans_28/train-00096-of-00188.parquet'
 'zho_Hans_28/train-00134-of-00188.parquet'
 'zho_Hans_28/train-00164-of-00188.parquet'
 'zho_Hans_28/train-00103-of-00188.parquet'
 'zho_Hans_28/train-00000-of-00188.parquet'
 'zho_Hans_28/train-00136-of-00188.parquet'
 'zho_Hans_28/train-00039-of-00188.parquet'
 'zho_Hans_28/train-00035-of-00188.parquet']
```

### English 

```
Selected directory: eng_Latn_45
Selected data files:
['eng_Latn_45/train-00190-of-00224.parquet'
 'eng_Latn_45/train-00198-of-00224.parquet'
 'eng_Latn_45/train-00031-of-00224.parquet'
 'eng_Latn_45/train-00064-of-00224.parquet'
 'eng_Latn_45/train-00002-of-00224.parquet'
 'eng_Latn_45/train-00185-of-00224.parquet'
 'eng_Latn_45/train-00202-of-00224.parquet'
 'eng_Latn_45/train-00009-of-00224.parquet'
 'eng_Latn_45/train-00081-of-00224.parquet'
 'eng_Latn_45/train-00048-of-00224.parquet']
```

### Greek

```
Selected directory: ell_Grek_4
Selected data files:
['ell_Grek_4/train-00178-of-00305.parquet'
 'ell_Grek_4/train-00187-of-00305.parquet'
 'ell_Grek_4/train-00194-of-00305.parquet'
 'ell_Grek_4/train-00042-of-00305.parquet'
 'ell_Grek_4/train-00067-of-00305.parquet'
 'ell_Grek_4/train-00224-of-00305.parquet'
 'ell_Grek_4/train-00097-of-00305.parquet'
 'ell_Grek_4/train-00007-of-00305.parquet'
 'ell_Grek_4/train-00114-of-00305.parquet'
 'ell_Grek_4/train-00290-of-00305.parquet']
```

### Hebrew

```
Selected directory: heb_Hebr
Selected data files:
['heb_Hebr/train-00082-of-00226.parquet'
 'heb_Hebr/train-00053-of-00226.parquet'
 'heb_Hebr/train-00217-of-00226.parquet'
 'heb_Hebr/train-00035-of-00226.parquet'
 'heb_Hebr/train-00027-of-00226.parquet'
 'heb_Hebr/train-00048-of-00226.parquet'
 'heb_Hebr/train-00114-of-00226.parquet'
 'heb_Hebr/train-00065-of-00226.parquet'
 'heb_Hebr/train-00134-of-00226.parquet'
 'heb_Hebr/train-00208-of-00226.parquet']
```

### Turkish

```
Selected directory: tur_Latn_4
Selected data files:
['tur_Latn_4/train-00055-of-00215.parquet'
 'tur_Latn_4/train-00165-of-00215.parquet'
 'tur_Latn_4/train-00100-of-00215.parquet'
 'tur_Latn_4/train-00049-of-00215.parquet'
 'tur_Latn_4/train-00004-of-00215.parquet'
 'tur_Latn_4/train-00016-of-00215.parquet'
 'tur_Latn_4/train-00047-of-00215.parquet'
 'tur_Latn_4/train-00127-of-00215.parquet'
 'tur_Latn_4/train-00169-of-00215.parquet'
 'tur_Latn_4/train-00054-of-00215.parquet']
```

### Vietnamese
```
Selected directory: vie_Latn_1
Selected data files:
['vie_Latn_1/train-00050-of-00094.parquet'
 'vie_Latn_1/train-00078-of-00094.parquet'
 'vie_Latn_1/train-00060-of-00094.parquet'
 'vie_Latn_1/train-00049-of-00094.parquet'
 'vie_Latn_1/train-00057-of-00094.parquet'
 'vie_Latn_1/train-00074-of-00094.parquet'
 'vie_Latn_1/train-00068-of-00094.parquet'
 'vie_Latn_1/train-00085-of-00094.parquet'
 'vie_Latn_1/train-00081-of-00094.parquet'
 'vie_Latn_1/train-00069-of-00094.parquet']
```