{'fraction_non_alphanumeric': 0.0199203187250996, 'fraction_numerical': 0.0, 'mean_word_length': 5.618421052631579, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Zipf's law, hyperbolic distributions and entropy loss

Zipf's law is an empirical observation which relates rank and frequency of words in natural languages. The law suggests modelling by distributions of "hyperbolic type". We present a general definition and an information theoretical characterization of such distributions. This leads to a property of stability and flexibility, explaining that a language can develop towards higher and higher expressive powers without changing its basic structure.