{'fraction_non_alphanumeric': 0.01451378809869376, 'fraction_numerical': 0.011611030478955007, 'mean_word_length': 4.948275862068965, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards 100M Morphologically Annotated Corpus of Tajik

The paper presents a work in progress: building morphologically
annotated corpus of Tajik language of the size more than 100
million tokens. The corpus is and will be by far the largest
available computer corpus of Tajik: even its current size is
almost 85 million tokens. Because the available text sources
are rather scarce, to achieve the goal also the texts of a
lower quality have to be included. This short paper briefly
reviews the current state of the corpus and analyzer, discusses
problems with either “normalization” or at least categorization
of low quality texts and finally also the perspectives for the
nearest future.