{'fraction_non_alphanumeric': 0.014120667522464698, 'fraction_numerical': 0.0, 'mean_word_length': 5.964285714285714, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Language Classification in Multilingual Documents

We investigate the use of machine learning techniques for language classification in a multilingual setting. We consider three contexts in which this task may be performed: identifying the language of monolingual documents, identifying the language of individual tokens, and finally identifying and correctly classifying spans of monolingual text in multilingual documents. Our broad goal is to achieve the third task with optimal efficiency and accuracy. We pursue this goal by building and examining various morphologically-based classification methods that attempt to classify, within a document, the language identity of individual words whose language is not known, thereby approximating a potentially multilingual setting.