{'fraction_non_alphanumeric': 0.013862024500322373, 'fraction_numerical': 0.0012894906511927789, 'mean_word_length': 4.92175572519084, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Retrieving a user language model from an unsupervised document map

This work presents a method to automatically retrieve a statistical language model focused on the specific topic and style of the speech situation at hand. The retrieval is based on a feature vector representation [2] of a sample text or an approximative intermediate speech transcription. This so-called document vector will be compared to an index of pre-trained language models and the best models are retrieved. The index is not just a list of language models but itself a smooth topological representation of language topics and styles found in all processed training material. The index is organized in a fully unsupervised manner by applying the Self-organizing map algorithm [1] to the document vectors obtained from all training documents [2]. The self-organization drags similar documents close to each other and gradually forces a local ordering in the map. In addition to smoothing the document reference vectors with their neighbors, the organized structure of the index can be utilized in other ways, as well. First, if the index is very large, the search for the best-matching reference can be boosted by performing a local search instead of a global one. In this way we can start the search based on any hint, such as the best match in the previous search, and proceed to search to its neighbors by following the topological structure defined by the index map. Another and even more important advantage of the ordered index is in the construction of the language models. While the set of documents pointed by a single reference in the index may sometimes be too small for creating a proper statistical language model, it is easy to increase the training set of relevant documents by including the sets from the neighboring indices, as well. The focused language model is formed by interpolating the prototype language models retrieved from the index. Because the topic and style of the language sample can seldom be exactly specified, or it may contain several topics, or even fall between the modeled topics, the index can be set to point to language model prototypes trained on different amounts of neighboring indices. The interpolation of the probabilities that the retrieved language models provide may also include a general model for the whole data as suggested in [3]. This ensures an adequate model for such rare events that may have been missed by the small topical language models. The suitability of the chosen overlapping language models can be evaluated by measuring the modeling accuracy by perplexity or word entropy on the sample text or the initial speech transcription. An example application of this language model structure is a multipurpose speech recognition tool that adapts to the user and speaking situation by operating in several passes over the speech data. In addition to model adaptation, the multipass recognition approach provides efficient means to enhance the recognition speed. The first pass can be performed with approximative general models and the second pass can concentrate on