{'fraction_non_alphanumeric': 0.013812154696132596, 'fraction_numerical': 0.0055248618784530384, 'mean_word_length': 5.415929203539823, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Language identification for transliterated forms of Indian language queries

Language identification has a number of applications in natural language processing. N Gram analysis has been the traditional method for language identification. In this paper, we discuss the methods and results from our participation in the Shared Task on Transliterated Search track at Forum for Information Retrieval Evaluation, 2014. We describe a method that leverages the phonetical properties of query tokens and their effect on the grammatical construction of tokens. The system developed is found to have a token level language labelling accuracy of around eighty percent for each of the language pairs, Hindi-English and Kannada-English.