{'fraction_non_alphanumeric': 0.029433903496249383, 'fraction_numerical': 0.010174682690061516, 'mean_word_length': 4.809408926417371, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Enhance to Read Better: A Multi-Task Adversarial Network for Handwritten Document Image Enhancement

Handwritten document images can be highly affected by degradation for different reasons: Paper ageing, daily-life scenarios (wrinkles, dust, etc.), bad scanning process and so on. These artifacts raise many readability issues for current Handwritten Text Recognition (HTR) algorithms and severely devalue their efficiency. In this paper, we propose an end to end architecture based on Generative Adversarial Networks (GANs) to recover the degraded documents into a clean and readable form. Unlike the most well-known document binarization methods, which try to improve the visual quality of the degraded document, the proposed architecture integrates a handwritten text recognizer that promotes the generated document image to be more readable. To the best of our knowledge, this is the first work to use the text information while binarizing handwritten documents. Extensive experiments conducted on degraded Arabic and Latin handwritten documents demonstrate the usefulness of integrating the recognizer within the GAN architecture, which improves both the visual quality and the readability of the degraded document images. Moreover, we outperform the state of the art in H-DIBCO challenges, after fine tuning our pre-trained model with synthetically degraded Latin handwritten images, on this task.


Introduction
Handwritten document analysis is an active and important field in computer vision and pattern recognition community. With the recent developments in machine learning [1], processing handwritten documents is reaching a good accuracy, especially in the application of Handwritten Text Recognition (HTR). HTR is the crucial part towards automatically understanding a document, which facilitates the access to various automatic applications such as: information extraction, search, indexation, validation, etc.
One of the problems that HTR systems are facing is the degradation of an inputted document, which significantly decreases the reading performance, reflecting on its utility. Indeed, many degradation scenarios can be attached to a handwritten document, especially the historical ones. Degradation includes background noise, corrupted text, dust, wrinkles and historical effects just to name a few related to the condition of the document itself [2]. The bad scanning process can also produce problems (shadows, blur, light distortion, angle, etc.) [3,4]. Moreover, some documents contain watermarks or stamps inserted for security reasons, those can cover the text and obstruct the HTR engine [5]. Some degradation examples are presented in Figure 1, as it can be seen, cleaning the document before passing it to the HTR stage should be done.
This cleaning task, called document enhancement, includes different recovering techniques, to reverse the degradation effect, for example: Binarization, dewarping, deblurring, watermark removal, etc. Classic recovery techniques were integrating image processing algorithms to be used as a filter that separates the degradation from the text. However, those methods are failing in removing the high degradation. Also, their parameters are usually set depending on the quality state of the addressed document to produce an optimal result. Thus, a manual intervention is needed in some cases to adjust the parameters, which is quite costly.
Given this, some modern document recovery techniques are appearing, using machine learning tools. Those are training deep learning models, mainly Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs), to learn the parameters for a direct mapping of any degraded document image into a clean binary version (without a restriction on degradation level) [6,7,8]. Similar to those, we proposed in [5] a document enhancement model called DE-GAN. However, and despite the high accuracy that we achieved in various enhancement tasks (Binarization, cleaning, watermark removal, deblurring), an important evaluation was not done. In fact, the goal of document enhancement is to provide a cleaner version of the image which is highly beneficial for HTR engines. But, in the mentioned approaches (including ours), the evaluation was conducted using only the visual similarity measurement between the recovered image and the Ground Truth (GT) clean version with some metrics that depend on pixels values. Thus, a HTR evaluation (means, passing the images to a HTR engine and compare the recognized text with the GT) is missing, for a better validation of the developed approaches. Also, these models are generally trained using only the images, while ignoring the text. As result, a model can easily evolve to deteriorate the text while cleaning the degraded image.
Motivated by those challenges, we propose a new method consists in an improved version of our previously developed DE-GAN, which was designed to recover the handwritten document to a clean version while ensuring its readability. Our approach is a deep learning model based on GANs that learns its parameters not only from the handwritten images pairs (degraded + GT), but also from the associated GT text. For this aim, we propose to add a recognizer that is trained jointly in a GAN model to assess the readability of the recovered document image. Hence, the model shall learn the best mapping of the degraded image to be as clean as possible while keeping its text readable. To accomplish this, and since the used datasets for document binarization does not (or rarely) contain the text information, we used two publicly available handwriting text images datasets (KHATT for Arabic script and IAM for Latin script) that are originally used for HTR to create degraded versions from the GT clean text lines images. The contribution of this paper can be summarized as follows: • To the best of our knowledge, this is the first work that integrates a recognition stage in a document binarization model. Thus, the degraded handwritten document will be recovered while maximising its readability, simultaneously. This is done by combining the GAN and the Connectionist Temporal Classification (CTC) losses functions: We eliminate the noise while preserving the handwritten text strokes.
• We demonstrate that training the recognizer progressively (on images ordered from the degraded domain to the clean versions), improves the recognition performance.
• The proposed model is simple, and flexible to restore different forms of degradation, independently of the document language. This was shown by the experiments conducted on two created datasets namely degraded-IAM (Latin script) and degraded-KHATT (Arabic script).
• We achieved the SOTA performance in handwritten document binarization according to H-DIBCO benchmarks.
The rest of this paper is organized as follows. Section 2 provides a review of prior works on document enhancement, in particular for document recovery and binarization. Then we present our proposed model in Section 3. After that, experimental results and comparisons with recent methods will be described in Section 4. Finally, a conclusion with a future direction is given in Section 5.

Related Work
This work aims to recover images that contain hard degradation by removing the background noise, while keeping its readability by HTR systems as accurate as possible. This application called document enhancement, is generally a preprocessing stage that produces an enhanced version of the document, in order to improve the recognition results of HTR engines [9].
Early methods known by global binarization, aimed to find a single threshold value for the whole document. A more sophisticated approaches, named local binarization, determine a different threshold value for each pixel. These threshold values are further used to classify the image pixels into foreground (black) or background (white). [10,11]. Nowadays, thresholding based methods are still evolving, for instance a global threshold selection method was introduced in [12] basing on fuzzy expert systems (FESs). In this method the image contrast is enhanced in a first step. Then, a pixel-counting algorithm is used with another FES for thresholds adjustment as a range, before choosing the right value from the middle of that produced list. Moreover, a support vector machine (SVM) based approach was proposed in [13]. Given the local features of the gray level images, degraded regions were classified according to SVM to be binarized according to one of four different threshold values. The main drawback of these thresholding methods is the sensitivity to the document condition, they usually fail to restore highly degraded images [2], since it is hard to obtain a good filter in those scenarios.
Later, energy based methods were introduced to track the text while binarizing the image. In [14], the ink was considered as a target to be maximized by an energy function, while minimizing the degraded background. Similarly, the background is estimated and subtracted from the degraded image using a mathematical morphology in [15]. However, the results using those hand crafted image processing algorithms were unsatisfactory.
Recently, deep learning architectures were used to tackle this problem by training their weights directly from raw data. In [16], the problem was formulated as pixels classification depending on sequences. Hence, a 2D Long Short-Term Memory (LSTM) was used to predict each pixel value whether belonging to the text or the degradation given a sequence of its neighbours. This process is, of course, time consuming. Thus, instead of classifying each pixel separately, images were mapped in an end to end fashion from the degraded version into the clean one using the Convolutional Neural Networks (CNNs). These architectures, called auto-encoders, lead to recent improvements in image denoising [17] and more particularly documents binarization [18,19,20] or deblurring problems [21]. This kind of applications are now called image-to-image translation, since the goal is to start from a degraded image and learn a mapping function that translate it into a clean domain. Following these approaches, [7] proposed an auto-encoder architecture that performs a cascade of pre-trained U-Net models [22] to learn the binarization with less data. Also, [23] proposed a neural network to learn the the enhancement/binarizatoin in an iterative fashion.
Other deep learning approaches modeled the problem as a generation task, where the goal is to generate a clean version of the image by conditioning on the degraded one. This process was carried out using GANs architectures, composed of a generative model that produce a clean version of the image and a discriminator to assess the binarization result. Thus, and motivated by other approaches where GANs significantly surpasses autoencoders in image-to-image tasks [24], some approaches applying this method were introduced. In [5], a conditional GAN approach was developed for document enhancement and achieved good results in recovering handwritten documents with several backgrounds degradation scenarios, it was also used for optical documents deblurring and dense watermarks removal. A similar cGAN based method was also proposed in [8], where the binarization performed by the generator was done in two stages, learning the pixels in different scales then combining the results to provide the final output. In [25], a strokes preservation method was developed using a GAN model, this was done by learning the text boundary in an auxiliary task for a better document binarization, especially with weak or ambiguous strokes. Another GAN's based method was proposed in [26] using a two networks frameworks, for document binarization. The first one was conditioning on the clean image to generate a degraded one, while the other network reconstructed the clean version conditioning on the degraded image. Thus, an unpaired data training was performed leading to good results when using the second network to binarize images. Similarly, [6] treated the problem as two stages: The first stage was devoted to augment the data by creating degraded handwritten images using a GAN model, while the second stage exploited the generated images to train an inverse problem binarization model.
To summarize, deep learning based methods are now significantly surpassing the classic or modern image processing handcrafted algorithms for the handwritten document binarization task. However, the only limitation that can be noticed is the "image only" based training. Because, the usual benchmarks for testing the binarization performance do not include a text recognition evaluation (a GT truth text of the degraded documents). Thus, those methods could easily delete some parts of the handwritten text while binarizing the image, without noticing. It is to note that we addressed this problem in a previous work [3], but for printed documents domain and using the Tesseract OCR engine to evaluate the produced text. Where the character error rate of the OCR on the generated image is inputted as an additional input channel to the discriminator. Contrary, we are proposing in this study, is a handwritten text recognizer that is jointly trained in the GAN architecture to maintain the text, while cleaning the degraded image. Thus, it is more flexible to be used for different handwritten languages and writing styles.

Proposed Method
We treat recovering a clean version from a handwritten degraded document as an image-to-image translation task using a generative model. Our GAN architecture is composed of three main parts, as shown in  A generator, a discriminator and a handwritten text recognizer. Since we are using the text information, the patches that are used during training should be in a readable form by a HTR, after binarization. Thus, the model is designed to be working at handwritten line images level. During training, the generator is conditioning on the degraded line image to generate the clean version. The generated image is passed to the discriminator to assess it as real (looks clean) or fake (looks degraded), for ensuring a realistic visually recovery. The image is also passed also to the HTR model to read it and compare the recognized text to the GT, hence, to maintain its readability while recovering it. The discriminator, as well as the recognizer, passed their feedback about the generated image through the adversarial loss. Noting that another additional Binary Cross Entropy (BCE) loss is integrated in the generator, for a faster convergence. In this way, the generator parameters are learned to produce a handwritten image that is as clean as possible, while keeping the text quality. In what follows, we explain the  three components presented in our architecture with more details.

Generator
Since we are doing an image-to-image translation process, the generator is designed as an auto-encoder model. We employ the U-net [22] for this task, in which the inputted image is encoded through a sequence of convolution layers with a down-sampling to reach a specific layer. After that, the image is decoded with a sequence of up convolution layers with an up-sampling. The model involves some skip connections after each two successive layers to recover images with lower deterioration, since the goal is to keep the text while removing the degradation. Thus, skip connections can help the decoder in maintaining the text features while producing the image. Figure 3 shows further details about the used generator. As can be seen, it is composed of 23 convolutional layers, with Dropout regularization and batch normalization layers. The output of this model is a single channel (in gray level) image, assumed to be the cleaned version of the inputted degraded image.

Discriminator
The discriminator is another Fully Convolutional Network (FCN) that produces an assessment of the generated image in term of visual similarity (pixel level) with the GT (real) images. The model was designed to take a degraded image with its clean version and output the class "real" if the clean version is the real GT, or assign the class "fake" if the clean image was produced by the generator. Both input images, which have of course the same size, are concatenated in an H × W × 2 shape. Then, the obtained volume is propagated in the discriminator model detailed in Figure 4, to end up in the last layer as a form of H/16 × W/16 × 1 matrix. During training, this matrix contains values that are equal to 1 in case of inputting the GT as a clean image, and equal to 0 in case of inputting the generator based enhanced image.

Handwritten Text Line Recognizer
The used handwritten recognizer is a Convolutional Recurrent Neural Network (CRNN) model, following the architecture presented recently in [27] and considered among the best HTR architectures. Noting that, any other HTR can be also used for this task, for instance: [28,29,30]. The model architecture is detailed in Figure 5. After enhancing the image by the generator it is inputted to an encoding stage that uses convolutional and gated convolutional layers, with integrated regularization techniques. The encoded image is passed later to the decoding stage, consists in two bidirectional Gated Recurrent Unit (GRU) layers. Finally, the CTC is used to decode the feature frames into text characters. The CTC layer is having the size of the character set plus one additional symbol corresponding to the blank symbol. During training, the recognizer could be fitted with two types of clean images, forming the following two scenarios: • S1: The Recognizer is trained at each iteration with the GT images  that are related to the degraded batch images inputted to the generator. The GT images are used with the associated GT text transcription for training in this process.
• S2: The Recognizer is trained using the images enhanced by the generator at each iteration with the associated GT text. The intuition behind, is that we may obtain a better recognition convergence that is going progressively from the degraded domain to the clean domain.

Training process
The different components presented above were trained jointly. The generator G, which is a function having the parameters θ G , is conditioned on the degraded image I d to provide its cleaned image that should be as close as possible to the GT image I gt . This image is passed to be validated by the discriminator D and handwritten recognizer R, with parameters θ D and θ R , respectively. D is giving an assessment of the cleaned image about its cleanliness to be Real or Fake, ). This adversarial training process of G and D can be formalized by: R is recognizing the generated image to maintain its readability with the CTC decoder, CT C(t, R θ R (G θ G (I d ))), where t is the GT text. Note that it is trained with a clean version of the image (whether using S1 or S2, presented above), at each same iteration: CT C(t, R θ R (I gt )). Also, for a faster convergence, a simple BCE loss is used in the generator between the cleaned images and the GT ones, BCE(θ G ). Thus, the generator is being affected by three factors to produce its generation. The whole architecture is formalized as: Where λ and β are the weights balancing the components intervention to produce the final generated image. During our experiments, we set λ to 1 and β to 10. For training, we used Adam's optimizer for the generator and discriminator components, while using the RMSProp for the handwritten text recognizer.

Experiments and Results
We provide in this Section the experiments that were done to validate the effectiveness of our proposed method. First, we start by presenting the metrics and datasets used in our evaluation.

Metrics
Following the usual approaches for handwritten document image binarization [31], we use the same metrics to validate the cleaned images. Those metrics which compare the images visual similarity with the GT clean ones, are: Peak signal-to-noise ratio (PSNR), F-Measure (FM), pseudo-F-measure (Fps) and Distance reciprocal distortion metric (DRD). In addition, since we are using the text information to validate our model, we utilise as well the HTR metrics for comparing the recognized text to the GT one. These metrics are based on the Levenshtein distance [32], and they consists in the Character Error Rate (CER) and the Word Error Rate (WER) measures.

Handwritten text databases
Usual handwritten document binarization databases do not contain text information [2,33,34]. Thus, we opt to create synthetically degraded images from the databases used in HTR tasks in order to exploit the GT text provided within these datasets. We address in this study two different alphabets: Arabic and Latin. From each, we took the most used database for handwritten text line image recognition: KHATT [35] and IAM [36], to add degradation. We call the created datasets, degraded-KHATT and degraded-IAM.

Degraded-KHATT
The KHATT dataset was developed for Arabic manuscript recognition, contains texts lines images with their associated GT texts. In our experiments, we used 6161 lines for training and a set of 1861 lines for testing, while a set of 940 lines was used for validation as it was done in [28]. Then, we added random distortions as shown in Figure 6 to obtain the degraded-KHATT dataset. To accomplish this, we insert different background images containing some flaws or artifacts. These background images are extracted mainly from public historical documents such as Nabuco, Bickley diary and Persian datasets. We have also applied different distortion operations, especially, dilation, erosion and blurs using random kernel sizes (2 × 2 and 3 × 3 for dilation, 2 × 2 , 3 × 3 and 4 × 4 for erosion and from 1 × 1 to 15 × 15 for blurring). We inserted also random vertical lines having random widths in order to simulate the noise that can occur in old historical documents.

Degraded-IAM
The IAM dataset was proposed for handwritten Latin script text recognition. It contains 8962 line images taken from the Lancaster-Oslo/Bergen (LOB) corpus. To insert degradation, we used same as in KHATT, 6161, 940, 1861 line images for training, validation and testing, respectively. We add dense backgrounds to simulate real historical deteriorated images same as it was done for the degraded-KHATT presented above. Examples of the obtained degraded-IAM are illustrated in Figure 7.

Arabic handwritten texts images recovery
For Arabic, we fed the proposed model with the training set of the created degraded-KHATT database. As stated above, the generator is trained to map the degraded image into a clean version, which will be evaluated by the discriminator and the recognizer. It is to note that in our experiments (for Arabic and Latin manuscripts), we used a high degradation for a meaningful evaluation in the hard scenarios. Also, we separate the background types between the training and testing sets (i.e there is no intersection in the background noise between the two sets). Table 1 illustrates the obtained results of the performed image binarization methods on the test set of the degraded-KHATT database. Reminding that we proposed two binarization scenarios sharing the same GAN architecture and integrating a CRNN recognizer, S1 and S2, stated before. The key difference between the two scenarios is the data fed to the recognizer during the GAN training stage. In scenario S1, the recognizer (we call it CRNN1) is fed with ground truth clean images, while it is fed with generated images (cleaned by the generator) in the second scenario (called CRNN2). As it can be seen, contrary to the previous related approaches, we evaluate the image in its visual quality (Binarization performance) and readability (recognition performance) at the same time. For readability, we tested each of our both scenarios S1 and S2 (Reco. CRNN1 and Reco. CRNN2) using the two recognizers (CRNN1 and CRNN2).
To compare our approach, we used a simple GAN architecture as a baseline. The architecture contains the same generator and discriminator of our architecture, but, without using a recognizer. We compare also with the method presented in [3] for printed text recovery, where an OCR is used during training as a part of the discriminator. However, since we are doing a handwritten text recognition (not optical text). We modify it by training a HTR having the same architecture as [27] to use it as a part of the discriminator, more details are given in [3].
Out of the results, we can see that using the GT images, a trained HTR engine based on CRNN [27] is reaching a CER of 12.04 % and 32.39 % as a WER, this is considered as our upper bound for recognition. Using the same model to recognize the degraded images, we obtain a poor performance of 91.18 % in CER, obviously because the model is trained on the clean data.
If we train the model on the degraded train set, it results in 30.34 % of CER and 54.44 % as WER. This experiment is done to verify later if we can surpass this performance (as a baseline) by cleaning the images then reading them, instead of training a model on the degraded domain.
The different binarization approaches, as it can be noticed, are enhancing the visual quality and the readability of the degraded lines. However, we can see that the baseline cGAN which is not taking the text into consideration while cleaning the image, is producing a result having a better visual quality in term of PSNR and DRD, but a worse readability compared to the methods integrating the text information. For the recognizer based methods, it is clear that our recovery method (S1) is leading to the best performance in terms of having a good visual enhancement while conserving  the text readability. Since by recognizing the produced images, we get a CER of 27.03 % and a WER of 52.84 % when using the recognizer of S1 and a CER of 24.33 % and a WER of 47.67 % when using the S2 recognizer. This proves that using the text information during binarizing the images is useful. Also, we notice that the progressive learning of a HTR (training in an order from the degraded images to their clean versions) in a multitask framework, is better for the recognition task. However, using the HTR pretrained with the clean GT images (as a separate task) during enhancing the document, is better for the binarization performance (visually). To illustrate our method's effectiveness, we show in Figure 8 and Figure 9 some qualitative results of recovering the distorted lines, ranging from easy distortions to the hard ones. We can see that our method is the most successful in producing clean images especially in cases of highly degraded ones. In fact, it can even recover the vanished handwritten text strokes. Furthermore, as we stated above and since different weights can be used in the recognizer loss level to control the enhancement, we perform an ablation study to evaluate the right trade of between the visual quality and the readability during enhancing. In other words, the effectiveness of the weight λ presented in Equation 2. This is done by varying the weight λ, then training the model with that setting and finally measuring the visual quality and readability (using the recognizer of S1) at each time to have the right option. The obtained results are shown in Table 2, where the experiments were carried out using the first scenario (S1) on a set of the Degraded-KHATT database, and ended up by selecting the setting of λ to be 1.

Latin handwritten texts images recovery
For Latin manuscript, we performed the same experiments using the degraded-IAM dataset. The obtained results are presented in Table 3. As it can be seen, training a CRNN [27] on the degraded images leads to 40.34 % as a CER and a WER of 74.05 %, with an obvious poor visual binarization quality since there was not a performed binarization with this way (using directly the degraded version). Contrary, by cleaning the image and passing it to the recognizer, better results were obtained. Here, same as the previous experiment, we are comparing our method to the basic GAN (without a recognizer), to validate the use of text information in our current method and our proposed one in [3]. It can be noticed that our method surpasses both GAN methods in the visual quality, and achieves the best text recognition rate compared to the other options. By using the recognizer trained in S1, we boost the CER by 1.50 % compared to [3], 5.46 % compared to the basic GAN and 14.29 % compared to reading handwritten images in the distorted domain. Moreover, using the recognizer trained in S2 we can even improve the CER result by 4.07 %. Furthermore, we show some qualitative results in Figures 10, 11 and 12, to visualize the performances of the different methods. Of course, reading the degraded image by a model trained on the GT clean images is not a suitable option. Also, training a model on degraded images is not improving the recognition, especially in the hard scenarios. That is why, enhancing the image and then reading it is the better solution. As it can be seen, our method is better in this practice especially than the baseline cGAN (without a recognizer), because ours is a text conservative method. Hence, it maps the image to a clean but readable domain, while the basic GAN is mapping the image to a visually clean version, without taking the text into consideration (see Figures 11 and 12).
For the sake of more confirmation and to prove that our model is independent from the used recognizer, we took a different state-of-the-art HTR that is Puigcerver's model [37] trained on the GT images of IAM and KHATT original datasets. Then, we carried a binarization stage to our degraded databases using different methods (including ours) and measure the final recognition performance. As it can be seen from Table 4, our proposed binarization method enhances the performance of the recognizer compared to the use of images binarized by the classic methods [10,11] or the recent cGAN's based one. Also, we can confirm the efficiency of our proposed binarization method compared to the baseline cGAN which did not integrate the text readability information. Table 4: Impact of the proposed binarization method (scenario S1) on the recognition performance by a HTR system.

H-DIBCO Competitions
After demonstrating the suitability of our proposed method in recovering clean and readable images from highly degraded ones. In what follows, we validate it in H-DIBCO competition on handwritten document binarization, using H-DIBCO 2012 [31], H-DIBCO 2016, DIBCO 2017 [39] and H-DIBCO 2018 [2]. Since our model was designed to enhance line images with a size of 128 × 1024, we binarize H-DIBCO images in form of patches having the same dimensions. We compare with the recent state of the art approaches, the winners of the different competitions [31] and the classic binarization methods [10,11]. To clean H-DIBCO images, and since they are formed of Latin script text, we used our pretrained model on the developed degraded-IAM dataset. Two scenarios were investigated: Using the model directly to clean the images, or fine tuning it with a similar distribution before using it. For fine tuning, we used the other DIBCO and H-DIBCO versions [2] and the Palm-Leaf dataset [33]. It is to note also that since the DIBCO datasets are not holding the text information, we removed the recognizer component during fine tuning process, we have frozen also the batch normalisation layers of the generator and we trained the architecture for one only epoch to keep the learned knowledge of the degraded-IAM. During cleaning, we feed our model with the original degraded image in two forms: A normal condition and a vertically flipped version. Thus, we produce two instances of the recovered images. The flipped image is, then, re-flipped again to the normal condition. After that, a voting method is used to produce the final binarized image, by assigning zero to the pixel value (black) if it is indeed black in the two produced images by our model. We found that this led to a better result instead of using just one image condition.
We start our experiments with H-DIBCO 2012, the obtained results are given in Table 5. As it can be seen, our model leads to competitive results to the state of the art approaches, with a superiority in two metrics (PSNR and FM). However, we can see that the model proposed in [7] is better in term of F ps and DRD. Then, we tested our method on a more recent dataset which is H-DIBCO 2016 [40]. As presented in Table 6, our model gives the state of the art compared to all the methods in the three metrics PSNR, FM and DRD and in the overall average.  Next, we tested with DIBCO 2017, which contains a mix of handwritten and printed degraded documents images. The results in Table 7 show that our model is not superior in this dataset, but it is competitive with the best approaches. We note that our model performance was affected by the type of binarized documents in this dataset, which contain several printed documents, while our model is designed essentially for the handwritten text. Finally, we tested with the most recent H-DIBCO 2018 [2]. The results are shown in Table 8, where we compare with the most recent state of the art results, the winner of the H-DIBCO 2018 competition and the classic binarization methods. The performance of our model is superior than the different approaches in term of PSNR, FM, F ps and average score.
Out of the obtained results in the different datasets, we can say that the classic thresholding methods [10,11] have a moderate performance compared to the recent deep learning approaches. Also, we can notice that if our model using only degraded-IAM for training, does not reach a satisfactory result because there is a domain gap between the training and testing data. However, fine tuning our model with the similar datasets leads to the best performance compared to all the state of the art methods in H-DIBCO 2016 and H-DIBCO 2018. While having a competitive result with the best approach in H-DIBCO 2012 that is [7], where we obtain a superior PSNR and FM scores. We can conclude also that our model is more suitable for binarizing the handwritten images, since it was pre-trained on the developed degraded-IAM dataset before the fine tuning stage.
Finally, we show some qualitative results about the binarization performance in Figure 14 that demonstrates our method superiority compared to  the other ones in this task. Also, we provide the binarization result of other images from the H-DIBCO 2018 dataset in Figure 13 where we obtained images that are very close to the GT. Moreover, Figure 15 shows an example where our method can even complete the missing pixels (that are not existent in the GT image), to provide a more readable text. Original GT Otsu [10] Sauvola et al. [11] Souibgui et al. [5] H-DIBCO 2018 Winner [2] Kang et al. [7] Dang et al. [25] Ours

Dataset selection for the fine-tuning stage
Selecting the right dataset for fine-tuning will improve the binarization performance. Thus, in this section, we study the impact of the different datasets on the binarization process carried out on the H-DIBCO 2016. For all our experiments, we tested using the other variations of DIBCO and H-DIBCO (from 2009 to 2018), except DIBCO 2019 since it has different distributions in term of degradation and document types. Also, we tested including similar datasets which were developed for binarization task, namely, Palm-Leaf [33], Nabuco [43] and Bickley-diary [44]. Images contained in these datasets suffer from different kinds of degradation, such as water stains, ink bleed-through, and significant foreground text intensity. As it can be seen from Table 9, using our model trained only on degraded-IAM leads to poor results, thus, a fine-tuning stage is necessary. Using the H-DIBCO images for fine-tuning improves the performance with a slight superiority than using the DIBCO ones. This can be explained by the type of text, because our model is pretrained to binarize the handwritten text. However, using H-DIBCO and DIBCO at the same time is a better option. Because, DIBCO contains a useful types of degradation that can be learned to be cleaned by our model even with a printed text. Also, adding other datasets is sometimes useful, but in other times deteriorates the performance. This can be noticed when adding the Palm-Leaf dataset which improves the binarization, while adding the Nabuco or the Bickley-diary is leading the model to learn a non suitable parameters for H-DIBCO 2016. This can be justified by the similar domain (degradation and text types) between H-DIBCO 2016 and Palm-Leaf distributions, while it is different with the other datasets.

Conclusion
In this paper, we proposed an architecture for handwritten document binarization based on GANs. Our method recovers the degraded images while conserving its readability by integrating a HTR to evaluate the enhanced image in addition to the discriminator. To the best of our knowledge, this is the first approach that includes textual information when performing the recovery process of handwritten documents. Experimental results proved the effectiveness of the proposed model in cleaning extremely degraded documents. We proved also that training a HTR model progressively on the images binarized by the generator at each iteration leads to a better performance in CER and WER. We used in this paper a CRNN as a recognizer, but it is worth to note that by using other HTR architectures we may obtain a better recognition performances, since our method is flexible to integrate different ones. Moreover, we obtain the best performance compared to the state of the art in H-DIBCO benchmarks of degraded documents binarization.
Training supervised approaches using paired data (degraded image with its clean version and GT text) is costly, since it is really hard to obtain this kind of resources. That is why most of the datasets are synthetically made. As a future work, we intend to explore the use of unpaired data. In this way, the real images can be easily obtained (degraded images, and clean ones). Thus, our model could be trained to produce images that are for the discriminator as much real as possible (similar to the clean real documents) and for the recognizer as much readable and real as possible (the recognized text must have sense).