task_categories:text-generation task_categories:fill-mask task_ids:language-modeling task_ids:masked-language-modeling annotations_creators:no-annotation multilinguality:monolingual language:tt license:mit size_categories:10K<n<100K region:us tt crawl news azatliq
---
annotations_creators:
  - no-annotation
language:
  - tt
license: mit
multilinguality:
  - monolingual
size_categories:
  - 10K<n<100K
task_categories:
  - text-generation
  - fill-mask
task_ids:
  - language-modeling
  - masked-language-modeling
pretty_name: AzatliqCrawl
configs:
  - config_name: default
    data_files:
      - split: news_noisy
        path: news_noisy_*
      - split: news_clean
        path: news_clean_*
tags:
  - tt
  - crawl
  - news
  - azatliq
---

### Dataset Summary

AzatliqCrawl is a document-level dataset in Tatar language based on Azatliq newspaper.
There are two versions released: the **noisy** dataset, which has no filtering, and the **clean** dataset,  which has a variety of filters applied (language identification using fasstext BOW and deduplication using MinHashLSH with number of permutations equal to 128 and threshold equal to 0.9), though it naturally has a fair amount of noise itself. Each dataset is released in a document-level form.

To load and use dataset, run this script:

```python
from datasets import load_dataset

tt_crawl=load_dataset("veryrealtatarperson/tt-azatliq-crawl")
```