{'fraction_non_alphanumeric': 0.02199223803363519, 'fraction_numerical': 0.00258732212160414, 'mean_word_length': 5.344262295081967, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
N-Gram Based Statistics Aimed at Language Identification

As the web grows, language identification and text categorization in general is becoming an important issue. Web environment provides enormous amount of documents, often with textual and grammatical errors, which makes the task of identification uneasy. N-gram based approach, which is the basic method for text categorization, could be used with slight modifications to perform with more than 95% correct classification rate. Combined with other methods, like frequent words counting or unique tokens, it is possible to create efficient and reliable software tools for various purposes. In this paper, we discuss the ability of n-gram categorization to classify an unknown text with the highest possible certainty.