{'fraction_non_alphanumeric': 0.02526595744680851, 'fraction_numerical': 0.010638297872340425, 'mean_word_length': 5.845454545454546, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Language identification using discriminative weighted language models

In this paper, discriminative weighted language models are proposed to better distinguish between similar languages. Through parallel phone recognizers followed by language modeling (PPRLM) system in the first stage, two best candidates are hypothesized and then processed using discriminative language models. Experimental results show that, compared with the traditional one-pass language identification (LID) systems, the proposed two-pass method can greatly improve the performance without considerably increasing the computational costs. Tested on the evaluation set of the CallFriend corpus, the final system achieved an error rate of 14.90% on the 30s 12-way close-set task.