{'fraction_non_alphanumeric': 0.020134228187919462, 'fraction_numerical': 0.0019175455417066154, 'mean_word_length': 5.177514792899408, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Robust named entity detection using an Arabic offline handwriting recognition system

Text from Arabic optical handwriting recognition (OHR) systems can provide key indexing information. In particular, the text is rich in named entities (NEs) and detection of such entities is critical for search applications. Traditional approaches for detecting NEs in optical character recognition (OCR) output look for these NEs in the single-best recognition results. Due to the inevitable presence of recognition errors in the single-best output, such approaches usually result in low recall. Given that a lattice is more likely to contain the correct answer, we explore NE detection from word lattices produced by our Arabic handwriting recognition system. Since the improvement in recall is accompanied by a large number of false positives, we use confidence scores based on posterior scores to control precision. We show a 7% improvement in true detects for the same false acceptance rate on using lattices instead of 1-best hypothesis for NE lookup.