{'fraction_non_alphanumeric': 0.01544660846205507, 'fraction_numerical': 0.002686366689053056, 'mean_word_length': 5.535087719298246, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Automatic line and word segmentation applied to densely line-skewed historical handwritten document images

There exists a high interest in the digitization of handwriting historical documents, in the quest to preserve the cultural heritage of nations. In general, these manuscript images present new segmentation difficulties with respect to non-historical documents. The problems come from features such as paper aging, faded ink, back-to-front ink superposition or variable line skew, among others. This paper presents a methodology for detecting and extracting the text lines of images from complex handwritten historical documents. The proposed line segmentation algorithm is based on computing a binary transition map of the document and then extracting and refining the corresponding line regions through skeletonization. To improve the accuracy of line segmentation, a new graph-based splitting method to separate the touching lines is introduced. Once text lines have been segmented, we propose an algorithm based on mathematical morphology operators and position heuristics, to extract the component words on each text line. The robustness and accuracy of our approach was tested on digitalized pages of two complex historical document datasets: the correspondence of Nabuco and the family papers of Graham Bell. We have also successfully compared our algorithms to other general line and word segmentation algorithms presented at the ICDAR 2007 Handwriting Segmentation Contest.