{'fraction_non_alphanumeric': 0.01303538175046555, 'fraction_numerical': 0.0074487895716946, 'mean_word_length': 5.183908045977011, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Comparison of Equations Describing the Ranked Frequency Distributions of Graphemes and Phonemes

Abstract Frequency of occurrence of words in a language is well described by Zipf's law. However, Zipf's law does not describe the distribution of the letters or phonemes from which words are composed. Examination of 32 text corpora from 18 languages shows that both letter and phoneme frequencies are well described by an equation first developed by Yule and by a parameter‐free equation that also describes the distribution of DNA codons.