language:en license:cc-by-sa-3.0 size_categories:1M<n<10M format:parquet modality:text library:datasets library:dask library:mlcroissant library:polars region:us
---
dataset_info:
  features:
  - name: text
    dtype: string
  - name: pos
    sequence:
      sequence: string
  - name: id
    dtype: string
  - name: url
    dtype: string
  - name: title
    dtype: string
  - name: tokens
    sequence:
      sequence: string
  - name: ok_num_tokens
    dtype: bool
  - name: ok_num_pos
    dtype: bool
  splits:
  - name: train
    num_bytes: 74491455326
    num_examples: 6407814
  download_size: 23878052325
  dataset_size: 74491455326
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
language:
- en
license: cc-by-sa-3.0
---

# enwiki_tagged

enwiki_tagged is a dataset where part-of-speech tags are applied to the entire text from [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) (20231101.en).
Word tokenization and POS tagging were processed using [nltk.word_tokenize](https://www.nltk.org/api/nltk.tokenize.html#nltk.tokenize.word_tokenize) and [nltk.pos_tag](https://www.nltk.org/api/nltk.tag.pos_tag.html#nltk.tag.pos_tag) respectively.



## Column Descriptions:

- text (`str`): The original text from wikipedia.
- tokens (`list[list[str]]`): An array where the original text is split into sentences, and each sentence is further split into words.
- pos (`list[list[str]]`): An array of part-of-speech tags corresponding to each element in tokens column.
- id, url, title: All correspond to wikipedia.
- (ok_num_tokens, ok_num_pos: Missed deletion. All values are true.)