{'fraction_non_alphanumeric': 0.011178861788617886, 'fraction_numerical': 0.0020325203252032522, 'mean_word_length': 5.042944785276074, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Probability Analysis of the Vocabulary Size Dynamics Using Google Books Ngram Corpus

The article introduces a method for determining a rate of appearance of new words in a language. The method is based on probabilistic estimates of the vocabulary size of a large text corpus. Backward predicted frequencies of rare words are estimated using linear models that are optimized by the maximum likelihood criteria. This approach provides more accurate estimations of frequencies for the earlier periods; the lower the frequency of the word during the analyzed period, the higher the benefit. A posteriori estimates of the frequency probability of appearance of new words were used to clarify the vocabulary size for different years and rate of appearance of new words. According to the proposed probabilistic model, it was shown that > 30% of investigated English and Russian word were appeared in the language before the moment when they were identified in the Google Books Ngram Corpus.