{'fraction_non_alphanumeric': 0.011180124223602485, 'fraction_numerical': 0.0, 'mean_word_length': 5.2, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Language model acquisition from a text corpus for speech understanding

Speech understanding can be viewed as a problem of translating the input natural language of speech recognition results into an output semantic language. This paper describes automatic acquisition of a language model for translating natural language into semantic language from a text corpus using a stochastic method. The method estimates the co-occurrence probabilities of input and output grammar rules as a translation language model. Since the amount of text is limited, estimating a reliable language model is difficult. Therefore, we propose a method of concisely modeling input and output grammars in order to estimate a reliable translation model. Our method is shown to be effective by experiments using the ARPA ATIS task.