{'fraction_non_alphanumeric': 0.008610086100861008, 'fraction_numerical': 0.0, 'mean_word_length': 5.310077519379845, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A new lexicon optimization method for LVCSR based on linguistic and acoustic characteristics of words

This paper proposes a new lexicon optimization method to improve recognition rate of large scale spontaneous speech recognition. Occurrence count and length of a word has strong correlation with difﬁculty of recognizing the word. First, we investigate the relation and make a word correctness probability model. The proposed method optimizes the lexicon by making compound words or phrases step by step based on the word correctness probability model so as to improve the estimated recognition rate of the system. The optimization method is applied to a large scale Japanese spontaneous speech corpus. Experimental results show that the language model using the optimized lexicon improves the recognition rate.