{'fraction_non_alphanumeric': 0.034033559073459, 'fraction_numerical': 0.010740628909114437, 'mean_word_length': 4.72585823660163, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Evaluation of a Region Proposal Architecture for Multi-task Document Layout Analysis

Automatically recognizing the layout of handwritten documents is an important step towards useful extraction of information from those documents. The most common application is to feed downstream applications such as automatic text recognition and keyword spotting; however, the recognition of the layout also helps to establish relationships between elements in the document which allows to enrich the information that can be extracted. Most of the modern document layout analysis systems are designed to address only one part of the document layout problem, namely: baseline detection or region segmentation. In contrast, we evaluate the effectiveness of the Mask-RCNN architecture to address the problem of baseline detection and region segmentation in an integrated manner. We present experimental results on two handwritten text datasets and one handwritten music dataset. The analyzed architecture yields promising results, outperforming state-of-the-art techniques in all three datasets.


Introduction
Document Layout Analysis (DLA) is a wellestablished area of research that aims to extract the structure with which the information that the writer wanted to convey was recorded on paper. This includes the lines of text, the structural groups into which the lines are organized and their role in the document (paragraph, marginalia, title, etc.), the regions of non-textual information (illustrations, musical staff, graphs, etc.), and finally, the relationships between all the elements in the document (reading order, linked references, etc.).
It is important to notice that even with a perfect automatic text recognition system (TSR, which stands for OCR, HTR, or HMR) [6,23,2,25,16] the information that can be extracted automatically from the document without a proper DLA system is very limited. For instance, a perfect text line level transcription only is completed when all text lines are grouped into their respective structural groups (e.g. paragraphs) in reading order.
What is more, most state-of-the-art TRS are designed to handle only one line of text at a time [2,18,29], so a DLA system must pre-process the images in order to split the document into a set of text lines. Even in the case of the few modern TRS that are able to process a full paragraph at a time [12,4,30,3] a DLA system is needed to pre-process the document and split it into those paragraphs. If the document layout is simple enough, some of these systems might be able to handle full pages without any previous DLA, but they would fail to handle documents with complex layouts. Consequently, most modern DLA systems aim to address the text line segmentation problem (or baseline detection) [8,1,19] in order to feed a TRS an obtain the corresponding transcript. Although it is an important step in the process, normally it is not enough to arrange the text lines in their structural groups and move towards a complete transcription process.
In this sense other works propose to train new DLA systems to segment the document into a set of structural groups [1,8], so text-lines can be grouped later on. On the other hand, in [19,21] an integrated model is proposed to handle both tasks at the same time so the text lines become already grouped as part of the process. Despite the good results reported, the main drawback of both kind of systems is the inability to properly handle overlapping regions.
In general DLA sub-tasks, such as text line and region segmentation, can be seen as an object segmentation problem [9], where each region of interest in the document (textual and non-textual) is a different object in the image. For example, illustrations, paragraphs, text-lines and marginalia. Hence, some works draw upon the deep learning systems developed for general object segmentation to improve text line and region segmentation. For instance, in [32,24] a region proposal neural network is used for region segmentation on printed documents, while in [17,27] similar systems are used for region segmentation and text line segmentation on handwritten text documents. However, to the best of out knowledge, there is no analysis of this kind of architectures on the multitask setting of DLA in complex documents.
In this work, we analyze the use of a region proposal network architecture (Mask-RCNN [10]) to address the problem of multi-task DLA (integrated baseline detection and region segmentation) on complex documents. Several training scenarios are analyzed, both in handwritten text and music documents.
The paper is organized as follows. In Sec. 2 we describe the system under analysis in this work. Section 3 describes the experimental setup adopted, datasets and model hyper-parameters. In Sec. 4 we present and discuss out results, while final remarks and future research directions are discussed in Sec. 5. Finally, in Sec. 6 details to reproduce our experiments are provided.

Multi-Task Document Layout Analysis
Even though DLA may be considered a much broader and complex problem, in this work we focus on two of the most basic hierarchical levels of the structure of a handwritten document and its corresponding DLA sub-problems, namely baseline detection and region segmentation. Nevertheless, we do not address the remaining sub-problems such as the reading order or other relations between the elements of the document.
Limiting the DLA to these two sub-tasks allows us to address it as an object segmentation problem on images [9], similar to [19,21,17].
Many architectures have been proposed to address the object segmentation problem on natural images using deep learning. We recommend [31] for a comprehensive review of the most promising architectures.
In view of the good results proved by Mask-RCNN architecture [10] on natural images, region segmentation on printed documents [32,24] and historical documents [17,27], in this paper we analyze its effectiveness on multi-task document layout analysis on complex handwritten historical documents.
Mask-RCNN is a multi-stage architecture specifically designed for multi-task problems. First, a backbone network is used to extract a good feature representation of the input data. Then a Region Proposal Network is used to generate a set of rectangle boxes which likely contain objects of interest. Finally, a Prediction Heads network is used to perform a set of tasks on the objects of interest, for instance classification, bounding box regression and mask generation.
Here we briefly describe Mask-RCNN architecture, while implementing details are presented in Sec. 3 Backbone: The backbone network is composed of a convolutional network that extracts features from the input image. Commonly, the backbone is composed of blocks of ResNet [11], VGG [26] or similar architectures that extract features at the scale of the input image. Also, it can be combined with a featurepyramid network [14] to extract features at different hierarchical scales.
Region Proposal Network : It is another convolutional network that scans the feature maps generated by the backbone by an sliding window, simultaneously regressing region bounds and scoring the membership of each region to a set of objects classes. On each location of the sliding window, k region proposals are predicted, parameterized relative to a set of k reference boxes or anchors. Typically anchors span a range of scales and aspect ratios (e.g. 32, 64, 128 and 1:1, 1:2, 2:1).
Then each region proposal is filtered by its scores and a non-maximal suppression algorithm is used to handle overlaps. Finally, the region proposals that remain are called "regions of interest" (RoI) and passed to the next stage.

Prediction
Heads: also called Prediction Branches, are a set of neural networks (convolutional or not) designed to generate a task specific prediction for each RoI generated on the previous stage. For instance, the most common prediction heads for object segmentation are region labeling, region bounding box regression and mask prediction.
Region labeling classifies each RoI into a set of "object classes", for instance in DLA, a set of classes could be text-line, paragraph and marginalia. On the other hand, region bounding box regression maps each RoI to an adjusted box that better circumscribe the detected object. This new bounding box could differ from the anchor used to generate the RoI. Finally, the mask prediction head classifies each pixel of the RoI as belonging or not to the silhouette of the detected object.
It is worth noting that the ability of the architecture to make predictions for each RoI independently of the others, allows the model to naturally address cases where the regions or objects touch or overlap each other. This is a major improvement over Unetlike approaches [19,1,8] where overlapping objects are very difficult to handle.

Baseline detection
As mentioned before in this paper we aim to analyze the effectiveness of the Mask-RCNN architecture to handle baseline detection and region segmentation on handwritten historical documents. While region segmentation is straightforwardly addressed by Mask-RCNN architecture, baseline detection is not.
The baselines of a document are normally represented as a piece-wise linear curve and cannot be feed directly to Mask-RCNN framework. Instead we use use the polygon that enclosses each text line as the object to search for, in order to detect the baselines. If this polygon is not available on the ground-truth, it can be easily generated as a fixed amount of pixels above and belong the baseline. Then for each "textline" object detected by the model, we use the baseline detection algorithm presented in [19] to obtain the baseline itself.
As counter intuitive this could sound-we convert baselines to text lines, then to baselines and finally to text lines again in order to feed any text recognition system -this methodology give us the versatility to handle ground-truth labeled using the cheaper baselines instead of the cumbersome text line annotation [22].

Experimental Setup
Our experiments are designed to analyze the performance of Mask-RCNN on complex handwritten documents. To that end, we experiment on different datasets, different initialization methods and compare them to state-of-the-art methods for multi-task DLA.

Datsets
In order to estimate the effectiveness of the Mask-RCNN architecture on handwritten documents we select three publicly available datasets. Particularly, because we want to analyze the performance of the model in a multi-task set-up, all datasets are annotated not only at baseline (text line) level, but a region level as well.
Two of the datasets are composed of only text related regions (paragraphs, marginalia, ...). This kind of dataset is expected to be very difficult for any object detection model, because the boundary which separates a region and another could be very subtle, specially on handwritten documents (See Fig. 1 and  Fig.3). The other dataset (VORAU-253) is a music manuscript. It was selected in order to analyze the performance of the model on handwritten documents where the regions contains more than just text.

Oficio de Hipotecas de Girona:
the manuscript (OHG) is composed of hundreds of thousands of notarial deeds from the 18-19 centuries (1768-1862) [20]. Sales, redemption of censuses, inheritance and matrimonial chapters are among the most common documentary typologies in the collection. This collection exhibit a relatively complex layout, composed of six relevant region types; namely: pag, tip, par, pac, not, nop. An example is depicted in Fig. 1. In this work we use the same training and test set defined in [19], composed of 300 images for train-ing and 50 for test. The main characteristics of this dataset are summarized on Table 1.  [28]. The dataset text is written in Early Modern German by an unknown number of writers. The public dataset is composed of 400 pages (350 for training and 50 for validation); most of the pages consist of two or three regions with many difficulties for line detection and extraction.
The ground-truth of this dataset is available in PAGE format (see [28]) and annotated at baseline level and region level. Regions have been labeled using four different region types, namely: page-number, marginalia, paragraph and heading. The main characteristics are summarized on Table 2 and an example shown in Fig. 2.  Red=page-number, green=marginalia, blue=paragraph, pink=heading. Better seeing in color.

VORAU-253:
is a music manuscript of the Vorau Abbey library, referred to as Cod. 253, which was provided by the Austrian Academy of Sciences. It is written in German Gothic notation and dated around year 1450. This manuscript is interesting because of the complexity of its layout, where staff, text and decorations are intertwined to compose the structure of the document (see Fig. 3).
In this work we carried out our experiments using the same training and test subsets as in [21], composed of 128 images for training and 100 for test. The dataset contains three different layout elements: • staff : Represents the regions that contains a set of horizontal lines and spaces, each one representing a different musical pitch. This region type does not contain text lines. Hence, no baselines.
• lyrics: words that are sung, which appear below their corresponding staff, and other text in the document.
• drop-capital : a decorated letter that might ap-pear at the beginning of a word or text line.
Main statistics of the dataset are presented in Table 3, and an example of an annotated page can be seeing in Fig. 3.

Evaluation Measures
To the best of our knowledge, there is no common meaningful evaluation measure able to assess the results obtained in both tasks jointly, therefore we present a set of metrics for each task.
Region segmentation: we report metrics from semantic segmentation and scene parsing evaluations as presented in [15]: Mean Jaccard Index (mIoU): Frequency weighted Jaccard Index (f.w.IoU): where K is the number of different classes in the task, η ij is the number of pixels of class i predicted to belong to class j, and τ i the number of pixels of class i. Even though we present both metrics, we consider Mean Jaccard Index the most relevant one for the region segmentation task.
Baseline detection: we report precision (P), recall (R) and its harmonic mean (F1) measures as defined specifically for this kind of problem in [7]. Tolerance parameters are set to default values in all experiments. Notice that precision and recall are defined respect to the segments of the so called "normalized baseline" and not directly to the whole baseline (see [7] for details about measure definition, tolerance values and implementation details).

System setup
As explained before we use the Mask-RCNN architecture in all experiments. As backbone we use the first four layers of ResNet50 [11] combined with FPN [14]. In order to evaluate how dependent Mask-RCNN is to the amount and type of training data available, the model has been trained using the following strategies: Mask-RCNN-s: a model is trained from scratch for each dataset using only the data available of each respective dataset.
Mask-RCNN-A: a single model is trained from scratch using all the data available on the three datasets together.
Mask-RCNN-I: on this case the weights of the backbone were initialized using ImageNet dataset [5]. Then, a model is fine-tuned for each dataset using the data data available of each respective dataset.
Mask-RCNN-I+A: like in the Mask-RCNN-I case the weights of the backbone were initialized using ImageNet dataset. But, instead of three different models a single model is fine-tuned using all the data available on the three datasets together.
Mask-RCNN-P: the weights of the backbone were initialized using PubLayNet [32] dataset. Then, a model is fine-tuned for each dataset using the data data available of each respective dataset.
Mask-RCNN-I+P: like in the Mask-RCNN-I case the weights of the backbone were initialized using ImageNet dataset. Then, the model is fine-tuned using PubLayNet dataset. Finally a model is fine-tuned for each dataset using the data data available of each respective dataset.

Training:
we fine-tuning the models through 90K iterations using the respective train set. Initial learning rate were set to 0.02 with a multi-step learning rate scheduler at 40K, 60K and 80K iterations.
In the case of models trained from scratch the initial learning rate were reduced to 0.001 in order to prevent the model from diverge at the beginning of the training process.
We use Stochastic Gradient Descent optimizer and a loss function composed of categorical cross entropy loss (L RPN and L R ) for RPN classification and region classification branch, smooth L1 loss L bb for bounding box prediction and binary cross entropy loss L mask for mask prediction. The total loss is a weighted combination of these losses, i.e. L = λ RPN L RPN + λ R L R + λ bb L bb + λ mask L mask . In our experiments we set λ RPN = λ R = λ bb = λ mask = 1.
Batch size is set to 4 images in order to keep the model under hardware memory restrictions (we use only one Nvidia GeForce GTX 1080 GPU and 15GB of RAM).

Inference:
during inference we keep the number of RoI after nonmaximal suppression to 1000, then only the best m RoI sorted by score are feed to the Prediction Heads stage, where m is selected based on the maximum number of objects in a training image n for each dataset as m = max(100, n + 50).
After network inference the results are postprocessed to remove regions with a probability to belong to any class lower than 0.5. Then, normal regions are separated from "text-line" to apply the baseline detection algorithm as discussed in Sec. 2.1. Finally, "text-lines" and their respective baselines are inserted hierarchically into the region with maximum IoU.

Results
Results of basic training configurations (Mask-RCNN-{s,I}) along with state-of-the-art results available are presented for comparison in Table 4.
In general Mask-RCNN is able to obtain very good results in all three datasets, surpassing state-of-theart methods for Mask-RNN-I, where the backbone weights were initialized using a pre-trained model. Models trained from scratch using only the training data of the dataset itself (Mask-RCNN-s) performs worse than the pre-trained counterparts. Likewise, those models are not able beat state-of-the-art models such as [19,8,21], which are fairly comparable to Mask-RCNN-s because they use only the training data of each dataset.
On the other hand, pre-trained model surpasses state-of-the-art models in all cases. It is important to notice that the region segmentation metric (mIoU) is specially improved by Mask-RCNN-I model. This is because Mask-RCNN is able to handle overlapping and touching regions, while previous models could not. For instance, in the Fig. 4 we can see an example of two touching regions that are well handled by a Mask-RCNN model, but merged into a single one by a model trained using the architecture proposed in [19].   Table 5 all trained models are presented for comparison 1 . Similarly to Mask-RCNN-I model, most of the pre-trained models achieve stateof-the-art results or better.

Furthermore, in
With respect to the models initialized using the PubLayNet dataset (Mask-RCNN-P), which is composed of printed documents, we observe no improvement of the results with respect to a general initialization using the ImageNet dataset (composed of natural images). Even in the cases where a tandem initialization was used (Mask-RCNN-I+P) the improvement observed is very modest.
Mask-RCNN-A and Mask-RCNN-I+A are interesting cases. In both cases all three datasets were merged into a single bigger dataset. Recall each dataset is composed of a disjoint set of regions. For Mask-RCNN-A we observe a degradation of the results with respect to the models where the datasets are used separately. This is mainly because some of the RoI's generated by the model belong to a class of another dataset (e.g. a region of type staff is assigned to a page of the OHG dataset). Besides, VORAU-253 dataset is an exception, mainly because the Mask-RCNN-s results are very low due the lack of training data (only 128 images).
In contrast, Mask-RCNN-I+A is able to obtain very good results (albeit statistically indifferent from Mask-RCNN-I) without any mislabeling observed between datasets. This is an important result that tell us about the versatility of the model, since we can train a single model and feed it with all incoming data without splitting it into the dataset it belongs to 2 . Also, from the point of view of processing large collections is important to reduce the number of models (i.e. the amount of memory used for them) in order to process the data efficiently and without compromising the results obtained.

Conclusions
In this work we analyze the effectiveness of models based on Mask-RCNN to handle DLA in a multitask manner (baseline detection, region segmentation and region classification). In general we have found that Mask-RCNN models need a lot of training data to achieve state-of-the-art results, however transfer learning helps to overcome this drawback. We obtain very good results using pre-trained models with the ImageNet and PubLayNet datasets, nevertheless better results were obtained using the first one.
Furthermore, experimental evidence indicates that the Mask-RCNN model can handle several datasets at the same time without compromising the effectiveness of the model. This important experimental result shows the versatility of the model and how it could be used to handle large heterogeneous collections.
In this work we analyze only one architecture based on region proposals. Future work will consider additional architectures to analyze the relationship between complexity and effectiveness. Equally important is to modify the Prediction Heads module to handle the relationship between layout elements, for instance the reading order and entity linking.

Reproducibility
Details required to reproduce our experiments are given below.  are publicly available in our online repository https: //github.com/lquirosd/ODDLA.
Dataset splits: we used the same dataset splits defined in previous works in order to keep comparability between experiments. OHG data splits can be found in the repository given in [19] 4 . For the Bozen dataset we used the same splits defined by the authors of the dataset [28]. Finally, we adopted the same data splits defined for the VORAU-253 dataset in [21], which is available in our repository as well 5 .
Hardware: we ran all our experiments on one GPU (Nvidia GeForce GTX 1080) and an Intel(R) Core(TM) i3-6100 CPU with 16GiB of memory. The hardware setup is important on this case because the Mask-RCNN architecture used in the experiments is 4 https://github.com/lquirosd/P2PaLA/tree/master/ egs/OHG 5 https://github.com/lquirosd/ODDLA/tree/master/data very memory demanding. For instance, datasets with very high number of regions of interest might not be possible to handle using this hardware setup.