{'fraction_non_alphanumeric': 0.045366795366795366, 'fraction_numerical': 0.005791505791505791, 'mean_word_length': 4.4867724867724865, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Entropy and Compression

The concept of entropy is fundamental to information theory. In 1948, Claude Shannon introduced entropy as a measure of the amount of choice in a set of events, where only the probabilities of the events are known [3]. Entropy can be understood in many ways, such as a measure of uncertainty, or a measure of the amount of information gained when learning the outcome of a set of events. Each interpretation is insightful to specific applications, such as compression and coding. Formally, for a discrete random variable X, entropy is the expected value of the information content of X. Thus, the entropy H of X is defined as H(X) = E[I(X)], if I(X) is the information content. The information content of an event is simply the negative logarithm of the probability of the event, I(p) = −logbp. This definition of information content has the benefit of being additive [3]. If p and q are two independent events, then the information content of the sequence (p, q) is I(p) + I(q). Thus, entropy is defined to be: