{'fraction_non_alphanumeric': 0.018726591760299626, 'fraction_numerical': 0.003745318352059925, 'mean_word_length': 5.1692307692307695, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A New Statistical Approach to Personal Name Extraction

We propose a new statistical approach to extracting personal names from a corpus. One of the key points of our approach is that it can both automatically learn the characteristics of personal names from a large training corpus and make good use of human empirical knowledge (e.g., Context Free Grammar). Furthermore, our approach also assigns confidence measures to the extracted personal names, compared with traditional simple true/false determination. Another main contribution of this work is that we have applied the personal name extraction technology into a real application, which is a Chinese inputting system and have achieved an approximately 7% error rate reduction for all characters and 30% error rate reduction for personal names.