{'fraction_non_alphanumeric': 0.0214168039538715, 'fraction_numerical': 0.006589785831960461, 'mean_word_length': 5.136363636363637, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Error recovery and sentence verification using statistical partial pattern tree for conversational speech

In this paper, in order to deal with the problems of disfluencies in conversational speech, a partial pattern tree (PPT) and a PPT-based statistical language model are proposed. A partial pattern is defined to represent a sub-sentence with a key-phrase and some optional/ functional phrases. The PPT is an integrated tree structure of the partial patterns generated from the training sentences and used to model the n-gram and grammatical constraints. In addition, a PPT merging algorithm is also proposed to reduce the number of partial patterns with similar syntactic structure by minimizing an objective cost function. Using the PPT, the undetected/misdetected errors due to disfluencies can be recovered. Finally, a sentence verification approach is proposed to re-rank the recovered sentences generated from the PPT. In order to assess the performance, a faculty name inquiry system with 2583 names has been implemented. The recognition accuracy of the system using the proposed PPT achieved 77.23%. We also contrasted this method with previous conventional approaches to show its superior performance.