{'fraction_non_alphanumeric': 0.02356902356902357, 'fraction_numerical': 0.0, 'mean_word_length': 5.538461538461538, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Statistical Stemming of Morphologically Rich Languages

We analyze current machine translation into Russian, a morphologically rich language, and present a technique for unsupervised statistical stemming. An initial pass is based on intuitions, and uses a CUDA kernel. As a later pass, we run EM. Since our model is relatively simple, the EM probabilities factorize into components that can be solved independently. While our results were not exactly grammatical, they appear to cluster grammatical classes, occasionally with some amount of lexicalization (extra characters preceding the word).