{'fraction_non_alphanumeric': 0.007122507122507123, 'fraction_numerical': 0.0, 'mean_word_length': 5.113043478260869, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Combining hidden Markov source models and parallel concatenated codes

We present here a framework for modifying a decoder for parallel concatenated codes to incorporate a general hidden Markov source model. This allows the receiver to utilize the statistical characteristics of the source during the decoding process, and leads to significantly improved performance relative to systems in which source statistics are not exploited. One of the constituent decoders makes use of a modified trellis which jointly describes the source and the encoder. The number of states in this modified trellis is the product of the number of states in the hidden Markov source and the number of states in the encoder.