{'fraction_non_alphanumeric': 0.021484375, 'fraction_numerical': 0.0, 'mean_word_length': 5.180722891566265, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Zipf's law and its modification possibilities

In this paper we consider the possibilities of known Zipf-Mandelbrot canonical law modifications. The proposed modifications explain the behavior of the right tail of the distribution and the presence of a deflection in the central part of the distribution (a crater). It is shown that the average word information load is invariant to the sample heterogeneity and that the proposed usage measure "places" the words more correctly with regard to their "importance".