{'fraction_non_alphanumeric': 0.01902398676592225, 'fraction_numerical': 0.011579818031430935, 'mean_word_length': 4.7075471698113205, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A thesaurus-based statistical language model for broadcast news transcription

This paper describes a thesaurus-based class n-gram model for broadcast news transcription. The most important issue concerned with class n-gram models is how to develop a word classification. We construct a word classification mapping based on a thesaurus so as to maximize the average mutual information function on a training corpus. To examine the effectiveness of the new method, we compare it with two our previous methods, in which the same thesaurus is used but word-class mappings are determined in different manners. The new method achieved substantially lower perplexity for 83 news transcription sentences broadcast on June 4, 1996. to build a vocabulary for broadcast news transcription. The size of the vocabulary in the experiment shown later is 20,000. We construct a word dictionary by adding to the vocabulary a word "UNK", which represents words not included in the vocabulary. Let V denote a set of words of the dictionary. Let V 1 denote a set of words included both in the thesaurus and the dictionary, and V 2 denote a set of words which are not included in the thesaurus but are included in the dictionary.