{'fraction_non_alphanumeric': 0.03447705663497313, 'fraction_numerical': 0.013600661430343117, 'mean_word_length': 4.629741680242029, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Auto-ML Deep Learning for Rashi Scripts OCR

In this work we propose an OCR scheme for manuscripts printed in Rashi font that is an ancient Hebrew font and corresponding dialect used in religious Jewish literature, for more than 600 years. The proposed scheme utilizes a convolution neural network (CNN) for visual inference and Long-Short Term Memory (LSTM) to learn the Rashi scripts dialect. In particular, we derive an AutoML scheme to optimize the CNN architecture, and a book-specific CNN training to improve the OCR accuracy. The proposed scheme achieved an accuracy of more than 99.8% using a dataset of more than 3M annotated letters from the Responsa Project dataset.


I. INTRODUCTION
The Optical Character Recognition (OCR) of printed media such as books and newspapers is of major importance, as it enables digital access, archiving, search and Natural Language Processing (NLP) based analysis of texts. A gamut of digitization projects, such as the America's Historical Newspapers, 1690-1922 1 , the California Digital Newspaper Collection 2 , the Doria project, and the digitization Project of Kindred Languages 3 , to name a few, were established. Some OCR projects deal with Hebrew language digitization, such as the Historical Jewish Press project 4 , and the Early Hebrew Newspapers Project 5 , while others, such as the Responsa Project 6 , Historical Dictionary Project 7 , Otzar HaHochma 8 and HebrewBooks 9 , focus on religious Jewish literature. It is common in such projects to utilize commercial OCR packages such as ABBYY 10  The Responsa Project is one of the largest scale digitizations of Hebrew and Jewish books. In particular, it specializes in Rashi script OCR that is an ancient typeface of the Hebrew alphabet, based on a 15th century Sephardic semi-cursive handwriting. It is has been used extensively in Jewish religious literature and Judaeo-Spanish books for more than 600 years, and is in large scale use nowadays. Rashi and Hebrew scripts letters are depicted in Table I. We denote as characters the different manifestations (in different fonts) of the same underlying letters.
Pattern Matching [1] and Feature Extraction [2], [3] were applied to OCR, by comparing series of image descriptors encoding the characters, such as SIFT [2], SURF [4], and PCA [3] to name a few. The recognition was formulated as a classification task using SVM [3], [2]. OCR was one of the first applications of Convolutional Neural Networks (CNNs), due to the seminal work by LeCun et al. [5].
The recognition accuracy of OCR can be improved by applying NLP to the visual analysis results. NLP was applied using dictionaries [6], [7], statistical algorithms such as HMM [8], Graph optimization [9] and LSTM [10]. Contemporary CNN-based NLP schemes often utilize word embeddings [11], alongside RNN and LSTM layers. The work of Kim et al. [12], is of particular interest as multiple stacked LSTM layers were used to compute a hierarchy of embeddings, first embedding the sequence of letters to represent words, and the sequence of words to represent sentences.
In this work we study the OCR of Rashi scripts that entails several challenges. First, being an exotic font, there are no commercial or academic OCR softwares for creating a training set. Second, it is common in some books to utilize the Rashi font as a primary font, alongside regular printed Hebrew fonts to emphasize the beginning of paragraphs, as shown in Fig. 1, or mark citations from older books, such as the Bible or the Talmud.
Third, the Rashi script books were printed by different printing houses, over more than 500 years, resulting in significant variations of the script, as depicted in Fig. 2 Some ancient Jewish books are fully or partially written in Aramaic language, contain special symbols for acronyms, and a rare composition of ‫א‬ (alf) and ‫ל‬ (lmd) as in Fig. 6. Implying that the OCR schemes can not apply standard dictionaries and spelling correction schemes, that have to be learnt as part of the OCR scheme. As the Rashi script consists of disconnected a b . characters, the agnostic detection of isolated characters, that is the detection of the bounding box of each character without detecting the character's class, can be easily implemented.
In this work we propose a Deep Learning based OCR scheme for learning both the visual representation of the Rashi script, and the spelling of the corresponding manuscripts. This is a particular example of scripts in an exotic language, where both the font and spelling are apriori unknown. The proposed scheme is fully data driven and does not utilize predefined dictionaries. We   derive a unified Deep network that combines a CNN to visually classify the characters, with a LSTM to learn the corresponding spelling, and thus improve the OCR accuracy.
In particular, we propose the following contributions: First, we derive a Deep network consisting of a CNN trained over 3M Rashi script samples, and a LSTM layer trained over words, that is shown to yield highly accurate OCR results.
Second, we derive an Auto-ML scheme based on Genetic Algorithms to optimally design the structure of the CNN. This results in an improved OCR accuracy achieved by a CNN three-fold smaller.
Last, we propose an active learning approach for refining the net accuracy when applied to a particular book, by refining the CNN model using a small set of the test characters.
The rest of this paper is organized as follows: we start by reviewing previous works on deep learning based OCR in Section II. The proposed approach is introduced in Section III, while the experimental validation and comparison between different schemes is presented in Section IV. Concluding remarks and future work are discussed in Section V.

II. RELATED WORK
Optical Character Recognition (OCR) is a common task in computer vision, used in a gamut of applications and languages such as, English [13] [14], Chinese [15] and Japanese [16]. The OCR of exotic languages such as the Rashi script is less applicative and common.
Similar to our work, Deep Learning (DL) was applied to exotic languages. A LeNet-based CNN [17] was applied by Rakesh et al. [18] for the OCR of the Telugu script consisting of ∼ 460 symbols that were encoded by binary image patches. A similar architecture was applied by Kim et al. [19] for the OCR of the handwritten Hangul script, while using the MSE as a loss function. A weighted MSE was proposed to handle the imbalanced training set during training.
A deep belief network (DBN), with three fully connected layers, was proposed by Sazal et al. [20] for the Bangla handwritten OCR dataset consisting of ten numerals and 50 characters. The layers were initialized using a restricted Boltzmann machine (RBM), and a softmax loss was used to classify the characters. A similar approach was applied by Ma et al. [21] for the OCR of a Tibetan script consisting of 562 characters and applied a three-layers DBM initialized using a RBM. SIFT local image descriptors were applied by Sushma and Veena [22] to encode and detect the characters of the Kannada language that uses 49 phonemic characters, where different characters can be composed to encode a single symbol. A Hidden Markov Model (HMM) was applied to improve the decoding of the language symbols by training the HMM using texts.
Deeper CNNs were used by Zhong et al. [23] for the Handwritten Chinese Character Recognition (HCCR) competition, by considering AlexNet [24] and GoogLeNet [25], where the images of the characters, their image gradients, HOG and Gabor descriptors were used as features. Stacked RNNs, the first for script identification and the second for recognition were applied by Mathew et al. [26] to the OCR of multilingual Indic Scripts, such as the Kannada, Bangla, Telugu and other languages. Both networks consisted of three hidden layers and a LSTM layer. Fuzzy logic and spatial image statistics were used by Gur and Zelavsky [27] to recognize distorted letters by using the combination of letter statistics and correlation coefficients.
OCR is a particular example of the Structured Image Classification problem where the classification problem is hierarchial. Such that the lower layers analyze the visual data, and the succeeding layers analyze the inner (semantic) structure of the data. In the proposed OCR scheme, the lower inference layer is implemented by a CNN that classifies the visual manifestations of the characters, and the semantic inference sub-network learns the particular dialect used in Rashi scripts using LSTM. A Structured Image Classification problem was studied by Goldman and Goldberger [28] in the context of structured object detection, where rows of products are detected in images by a CNN-based object detection scheme. The order (structure) of the detected objects is learnt by embedding the indexes of triplets of neighboring objects by an embedding layer, and concatenating the structure embeddings with the Fully Connected (FC) layer of the object detector. The resulting vector is used to infer the true label of the input image. In contrast, the proposed scheme processes a sequence of object images simultaneously, and the sequential data is encoded by a LSTM layer. Bar et al. detect compression fractures in CT scans [29] of the human spine. The spine that is a linear structure, was first segmented, and its corresponding patches were binary classified using a CNN. Recurrent Neural Network (RNN) was used to predict the existence of a fracture. Our proposed scheme infers multiple labels both by the CNN layers (characters) and LSTM (words).

III. OCR OF RASHI SCRIPTS USING DEEP LEARNING
We study the OCR of Rashi scripts where the locations of the characters are initially detected. Hence, the OCR is formulated as a classification problem, where given a sequence of images of characters {φ 1 , ..., φ J }, we aim to classify C(φ j ) ∈ {c 1 , ..., c L }, that are the latent character labels. We utilize a Deep Learning scheme consisting of two stacked sub-networks. The initial one, is a CNN that computes the detection probabilities of the characters based on the input images, while the succeeding sub-network utilizes a LSTM layer to improve the decoding accuracy, by learning a data-driven vocabulary. An overall view of the proposed scheme is depicted in Fig. 7. We studied several CNN architectures for classifying the images of the detected characters. The first denoted Simple NN (SNN) is based on the LeNet5 architecture [17] detailed in Table II.
Following the work of Zhong et al. [23], we also applied a deeper CNN based on AlexNet [24], detailed in Table III.
The Spatial Transformer layer [30][31] was shown to improve OCR accuracy, by geometrically rectifying  [24]. the input image by estimating the underlying affine transformation. Thus, we added the Spatial Transformer layer to the MNIST CNN, and denote the resulting CNN Spatial Transformer Net (STN), as reported in Table IV.
Thus, the output of the CNN is the classification probabilities, {p j,l } J,L 1,1 of the input sequence of character images {φ j } J 1 , such that p j,l is the estimated probability of an image φ j ∈ {φ j } J 1 to be related to the class (letter) c l ∈ {c l } L 1 . As a script might contain both Rashi and printed Hebrew fonts, we set L = 56, consisting of 27 characters of Rashi script, 27 characters of Printed Hebrew script, one for a special character 'a-l' and a single space character. The scripts are written in a particular Hebrew dialect that can be utilized to better infer the estimated symbols, as being part of words and sentences. For that we apply Long-short term memory (LSTM) [32] that allows to encode and utilize the inherent sequential information. The LSTM layer is denotes as L2 in Fig. 7

A. AutoML CNN architecture refinement using a genetic algorithm
We applied AutoML to refine the CNNs proposed in Section III by optimizing the classification accuracy with respect to the CNN architecture, that is given by the parameters of its layers. In particular, we consider a CNN similar to LeNet5 [17] consisting of sequential pairs of convolution and activation layers that are jointly optimized. Thus, we optimize the parameters of the convolution layers: the support of the filter, the number of filters (output dimensionality), the stride in both image axes, the use of batch normalization, and the use of a dropout layer and its probability. The activation layer is given by its type (ReLU, Avg-Pool, Max-Pool), where the pooling layers are given by their pooling support and stride.
AutoML is a nonlinear optimization over a heterogeneous set of discrete (filters support, stride etc.) categorical (activation type) and continuous (dropout rate) parameters. For that we apply a Genetic Algorithm (GA) [33] by first (generation g = 0) drawing N = 120 random CNNs CN N g=0 i similar to LeNet5, each consisting of l convolution+activation layers, followed by a fixed FC layer. Each such CNN is trained for M = 10 epochs using ∼ 5% of the training data. The This scheme is summarized in Algorithm 1, and was applied to generate multiple consecutive generations. It is initialized by running Algorithm 1 for l = {3, 5, 7, 9} and selecting the resulting CNNs having the highest validation accuracy. Algorithm 1 is then applied to this set of CNNs, and the resulting CNN is shown in Table  V.

B. Book specific OCR-CNN refinement
A particular attribute of the Rashi scriptures is that their Hebrew dialect was and still is a sacred religious dialect and was not in daily use, as the scholars using it lived in non-Hebrew speaking countries: Europe, North Africa, etc. Thus, it did not undergo significant changes in 600 years, and over different geographical regions. Hence, most Rashi manuscripts share a similar dialect and vocabulary, but might differ in the graphical printing attributes of the Rashi script, when printed by different printing houses, as depicted in Fig. 2.
In order to utilize the dialect invariance, in contrast to the varying graphical manifestation of the Rashi script letters, we propose to refine the CNN per script while freezing the LSTM layer, and use the test classifications as a training set for refining the CNN. For that, given a test manuscript, we apply the CNN+LSTM as in Section III, to ∼ 5% of the new script, and utilize the OCR output as a training set for refining the CNN. The resulting CNN is applied to the rest of the manuscript.

IV. EXPERIMENTAL RESULTS
In this section we experimentally verify the validity and accuracy of the proposed scheme by applying it to a large dataset provided by the Responsa project consisting of ∼2500 pages extracted from 170 different books. The dataset contains 5.5M annotated letters given their positions and labels. There are on average 3400 letters per page, where some of the manuscripts contain several pages, while others more then 20 pages. The images of the letters are of size of 60x30 pixels, and their distribution is depicted in Fig. 8. We divided the Responsa project dataset into three sets S r , S v and S t consisting of 3.3M, 352K, and 2M letters, respectively. S r is the training set consisting of 140 books and more than 1000 pages. S v was the validation set (114 pages from 15 books), and S t was the testing set collected from other 15 books. The test set consists of the first 40K letters in each book, while the set S t 2 comprises of the remaining letters in a manuscript, 1.35M letters overall. S t 1 was used for the book-specific CNN refinement scheme introduced in Section III-B, while S t 2 was used as a test set. We applied the CNNs detailed in Section III and the CNN computed using the AutoML scheme introduced in Section III-A, for which we used 2% of the samples in the set S r and 10% of S v , 75K and 35K letters, respectively. The AutoML refinement was applied using seven different initial 'families', each was trained for G = 10 generations and 120 mutations each. The number of convolution layers, the activations succeeding each convolution layer (ReLU, Max-Pool or Avg-Pool), and the addition of batch normalization or dropout layers are randomly drawn. Similarly, the layers' parameters (filters sizes, max pooling size, stride value and dropout percentage) are also randomly drawn. The families differ by the maximal number of initial layers.
Other than the first generation, all of the mutations were chosen randomly, the succeeding generations were composed of K 1 = 5 Elite group, 70 Cross-Over children and 25 mutations of the parents. The group of parents was collected from the K 2 = 40 most accurate CNNs (including the Elite group) computed in the pre-vious generation. We also added additional 20 random mutations at each generation to increase the variability. The Elite group from the last generation of each family is gathered into a single set of CNNs and refined for G = 10 generations and 120 mutations. Figure 9 depicts the average classification error reduction with respect to the number of training AutoML generations. Valid -Characters Test -Characters Fig. 9. The classification accuracy of the AutoML optimized CNN with respect to the Genetic Algorithm's refinement iterations.

A. OCR results using AutoML CNN
We start by evaluating the proposed OCR scheme accuracy of the four CNNs introduced in Section III, without the additional accuracy of the LSTM layer. The accuracy of the overall scheme (CNN+LSTM) is discussed in Section IV-B. Figure 10 reports the classifications accuracy of the CNNs with respect to the size of training set S r . For each CNN we consider the accuracy of detecting the 56 graphical symbols denoted as Characters, and the accuracy of detecting the set of 29 phonetic equivalents as Letters, where each Letter (other than 'space' and punctuation marks) can be mapped to two Characters, in Rashi and Printed fonts. The accuracy results are also reported in Table VI, where it follows that the SNN (based on the LeNet5 CNN) is inferior to the deeper AlexNet-based ALX and STN CNNs. The Spatial Transformer layer does not improve the accuracy significantly compared to AlexNet, and we attribute that to the deskewing of the OCR input images applied in the preprocessing phase, such that the additional rectification of the Spatial Transformer layer is insignificant.
The accuracy gap between the Characters and Letters is due to the similarity in Rashi and Printed Characters of some of the letters, as depicted in Table I. Thus, the CNNs might misclassify some character images as being Rashi instead of Printed and vice versa, while relating to the same Letter. This is considered a Character classification error, and a correct Letter classification. The proposed AutoML scheme is shown to significantly outperform all other schemes while utilizing the least number of the CNN parameters.   Fig. 11 list and depict the most misclassified Characters, and the most common misclassifications for each Character. It follows that the misclassified characters are indeed visually similar, and might be difficult to distinguish even for a human observer. Hence the need to utilize the sequential information using LSTM.

B. OCR results using AutoML CNN+LSTM
Following the results of Section IV-A, we studied refining the performance of the AutoML-based CNN by  adding a LSTM layer to the AutoML CNN to learn the Letters sequences. For that we considered three training strategies. First, we "froze" the CNN weights, (L1 in Fig. 7) and trained only the LSTM layer. Denote this CNN as AutoML-LSTM. Second, we further refined the AutoML-LSTM network, by unfreezing its CNN weights (L1 in Fig. 7) and "freezing" the LSTM and FC layers (L2 and L3) and denoted the resulting network RE-CNN. Last, we refined the AutoML-LSTM CNN by training the entire network (L1+L2+L3 in Fig. 7), and denoting the resulting network RE-BOTH. Figure 12 reports the classification error of the three schemes, and compares them to the CNN-only schemes discussed in Section IV-A. It follows that the AutoML-LSTM scheme significantly outperforms all other training variations, and the CNN-only schemes.   Fig. 7) layers and "freezing" the CNN layers (L1 in Fig. 7) RE-CNN: trained by "freezing" the LSTM and FC (L2+L3 in Fig. 7) layers and training the CNN layers (L1 in Fig. 7). RE-BOTH: training the entire network (L1+L2+L3 in Fig. 7).

C. Book-Specific OCR Refinement
The book-specific refinement scheme was introduced in Section III-B, allowing to utilize the invariance of the dialect of the Rashi scripts in contrast to the varying graphical manifestation of the characters. For that, given a particular test script (book), we applied the AutoML-LSTM CNN, trained as in Section IV-B, on a small subset of the test script, and used part of these test results, to refine the AutoML-LSTM CNN.
Thus, for each test script we applied both the CNN layers (L1 in Fig. 7) of AutoML-LSTM, and the entire AutoML-LSTM (L1+L2+L3 in Fig. 7). The characters that were classified similarly by both schemes were used as a refinement set for the CNN layers of AutoML-LSTM. Tables IX and X show the error classification rates of the proposed refinement scheme for the Characters and Letters, respectively. As the refinement set was relatively small, we repeated the experiment four times.
It follows that the proposed refinement scheme reduces the Letters classification error of the AutoML CNN+LSTM scheme from 0.188% (in Table VI) to 0.164 (in Table X), exemplifying the validity of the proposed scheme. We note that the refinement parameters corresponding to the highest Characters classification error (0.611% in Tables IX), do not correspond to the highest letters classification accuracy (0.164% in Tables X). We attribute that, as in Section IV-A, to the similarity of some of the Characters in both Rashi and Printed scripts, resulting in Characters misclassifications, but accurate Letters classifications. Tables VIII and XI, report the classification errors of the most confused letters, with and without the bookspecific refinement, respectively. The classification errors of most letters were improved, while the accuracy of the letters ‫ס‬ and ‫ם‬ ('smk' and 'mem-p') was not. The refinement scheme improves the CNN phase, and the visual inference, while ‫ס‬ and ‫ם‬ are difficult to distinguish visually. We show qualitative misclassification results in Table  XII, where some of the misclassification (examples 1, 2, 4, 5, and 14) are due to printing errors, while others (examples 6, 7 and 12) are due to paper erosion. Some of the misclassifications (3, 9. 10. 11 and 13 ) are quite similar visually and difficult to distinguish even by a human observer.

D. Implementation issues
The CNN (L1 in Fig. 7) and the LSTM (L2 + L3 in Fig. 7) layers of the proposed scheme were trained and implemented using the Keras wrapper [34] for Ten-sorFlow [35]. The proposed AutoML refinement scheme introduced in Section III-A, that is based on a Genetic Algorithm, was implemented in Matlab and MatConvNet [36]. All of the training was conducted on Titan X Maxwell and the training of the AutoML CNN required four hours.

V. CONCLUSIONS
In this work we proposed a Deep Learning based approach for the OCR of manuscripts printed in the Rashi and Printed Hebrew scripts. For that, we studied and compared four CNN architectures. In particular, we learn both the characters visual appearance and the (apriori unknown) vocabulary using CNNs and LSTM, respectively. We derive an AutoML scheme based on Genetic Algorithm to optimize the CNN architecture with respect to the classification accuracy. The resulting CNN is shown to compare favourably with other CNNs based on larger (AlexNet) or similar nets. We also propose a book-specific approach for refining the OCR CNN per book. The resulting scheme is shown to significantly outperform similar unoptimized CNNs and achieves OCR accuracy of 99.840%. OCR is a particular example of the Structured Image Classification problem that has multiple applications in computer vision [28] and medical imaging [29]. In future we aim to further develop and apply the proposed AutoML and book-specific refinement schemes in that context.