{'fraction_non_alphanumeric': 0.02366345311130587, 'fraction_numerical': 0.0, 'mean_word_length': 4.767676767676767, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Relationship of Machine Learning and Data Compression to Natural Language Processing : A Review

We review the state of the art in natural language processing (NLP), machine learning, and data compression. Central to both the NLP and the compression problems is that of finding a language model, or probability distribution P over the set of strings in a human language such as English. As large text corpora become available, the traditional NLP approach of explicitly specifying P as a set of rules is giving way to machine learning approaches, such as those used by compression algorithms. Still, we do not know how to accurately estimate P(x) when x is longer than a few words, a task that humans perform easily. Furthermore, we do not know if an algorithmic solution is even possible, and if so, its costs in memory, speed, or software development effort. We examine what is known about language and learning, both from the computational side of AI, where we try to duplicate human intelligence using any engineering technique available, and from the psychological side of AI, where we try to understand how the human brain does it.