{'fraction_non_alphanumeric': 0.026915113871635612, 'fraction_numerical': 0.0013802622498274672, 'mean_word_length': 4.870445344129554, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Review of variable-length codes for data compression by David Salomon

Any digital data—be it text, image, audio, video, or others—consists of a set of symbols called an alphabet. Each symbol of the alphabet is represented in the computer as a distinct pattern of bits (0 and 1). In the simplest representations—ASCII codes for example—each symbol is represented by the same number of bits. In practice, however, not all symbols occur with the same frequency. It is easy to observe that assigning shorter length codes to more frequent symbols and the longer codes to less frequent symbols leads to a save on space; this technique is called data compression. Any data compression scheme (also called coding scheme or simply code) consists of an encoder and a decoder. An encoder takes a symbol represented in bits (called dataword) as input and outputs a possibly shorter bit representation (called codeword) for that symbol. A decoder, on the other hand, does just the opposite; it takes codewords as input and outputs datawords. Any compressed data are, in fact, long sequence of concatenated codewords. Any such sequence must be uniquely decodable or uniquely decipherable by the decoder; that means the decoder must produce only one sequence of datawords for a compressed data. During a transmission, one or some bits of the data may get corrupted, hence a practical coding scheme must also have error detection and error correction capabilities.