{'fraction_non_alphanumeric': 0.03428571428571429, 'fraction_numerical': 0.0, 'mean_word_length': 4.841666666666667, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Estimating Frequency Functions from Limited Data

It is often necessary to estimate a frequency function or certain points on a frequency function from very limited data. A usual procedure for this estimation involves two steps. From the set of “well-known” frequency functions, e.g., the normal, poisson, binomial, etc., one chooses that function which seems likely to best “fit” and then uses the available data to estimate the parameters of the chosen distribution. If no one “well-known” function can be chosen a priori, then perhaps several likely candidates are tried and the one which fits best according to some criterion is chosen. For many purposes, this procedure is quite unobjectionable.