{'fraction_non_alphanumeric': 0.031958438287153654, 'fraction_numerical': 0.020465994962216624, 'mean_word_length': 4.65993465993466, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Optical Font Recognition in Smartphone-Captured Images, and its Applicability for ID Forgery Detection

In this paper, we consider the problem of detecting counterfeit identity documents in images captured with smartphones. As the number of documents contain special fonts, we study the applicability of convolutional neural networks (CNNs) for detection of the conformance of the fonts used with the ones, corresponding to the government standards. Here, we use multi-task learning to differentiate samples by both fonts and characters and compare the resulting classifier with its analogue trained for binary font classification. We train neural networks for authenticity estimation of the fonts used in machine-readable zones and ID numbers of the Russian national passport and test them on samples of individual characters acquired from 3238 images of the Russian national passport. Our results show that the usage of multi-task learning increases sensitivity and specificity of the classifier. Moreover, the resulting CNNs demonstrate high generalization ability as they correctly classify fonts which were not present in the training set. We conclude that the proposed method is sufficient for authentication of the fonts and can be used as a part of the forgery detection system for images acquired with a smartphone camera.


INTRODUCTION
Documents containing personal data are required for many services. The most obvious and potentially dangerous ID forgery occurs at the border control, where manual document inspection becomes almost impossible with constantly increasing passenger traffic [1]. When it comes to financial services [2], banks, insurance and telecommunication companies suffer huge losses. According to the Russian National Credit bureau, annual losses in the credit segment alone exceed 150 billion of Russian rubles [3]. In spite of all protection measures, the number of forged documents increases every year [4].
In the meantime, more and more companies develop services that require ID images captured with smartphones and other mobile devices [5]. Thus, the problem of automatic authentication of documents in photos in the visible spectrum becomes more and more relevant.
Most common type of document forgery (identity documents included), especially while using digital ways of data transmission, is text data falsification [6]. For a number of identity documents, particular fonts are specified by the government decrees and standards, for example, OCR-B font used within machine-readable zones [7]. Since a number of the special fonts are not publically available or officially classified, fraudsters use fonts that are similar but not identical to the specified ones. For instance, Fig.1 (a) demonstrates a line from a machine-readable zone (MRZ) of a counterfeit document printed with a forged font and Fig.1 (b) shows its version meeting the standards [7]. The aim of our study is to develop a neural network method to determine the conformance of a font in the document with a standard one using images of separate symbols captured with a smartphone camera. It is essential that the method should demonstrate adequate results on fonts which are used as the replacement for the one specified in the standards but are not presented in the training set. This paper is structured as follows: in Section 2 we provide an overview of forgery detection methods for documents in images and the methods of optical font recognition. In Section 3 we formulate a problem statement and describe its special traits. In Section 4 we describe train and test data and discuss the experiments.

Methods of detecting forgery of the documents based on images
In the previous study [1], the correspondence between data in the main part of the document and data in the machine-readable zone is analyzed, and also cross-verification is performed for images of the same class. Other works [8,9] propose detection methods for "copy-paste" type forgery, i.e. the authors use pixel analysis to detect the similarity among symbols that were copied and repositioned within the same document. Additionally, some authors suggest using the typographical features of fonts [10]. Based on their conclusions [10], other researchers propose to use Conditional Random Fields to get the estimation of text font by its typographical features and to assess its correspondence with other text within the document. However, previously applied methods cannot be fully embedded in a system for images acquired with small-scale digital cameras in uncontrolled conditions [12], as those approaches assume that images are captured with an even distribution of light [1], and with the resolution of no less than 300 dpi [6,8,9,10,11]. Although some authors [9] examine the possibility of image compression with different JPEG quality factors that mimics input data quality variability, it is not sufficient to imitate smartphone shooting conditions [12].

Method of optical font recognition
Fonts are usually recognized by character sequences (strings, words) without recognizing the characters themselves. Some authors suggest a method of font recognition from a set of more than 2000 classes of fonts using the nearest class mean classifier based on the local feature embedding [13]. Other studies demonstrate the efficient usage of multilayer CNN decomposition and SCAE-based domain adaptation [14]. In these works [13,14], the training data for the classifier is generated with known public fonts. In numerous studies focused on optical font recognition, the authors analyze exclusively high-resolution images, for example, with a minimal resolution of 400 dpi [15]. In studies [6,8], appliance of optical font recognition for forgery detection is suggested. Yet it is emphasized by the authors that noise in the images leads to "rapid decrease of recall" and that their method faces difficulty "to accurately classify each character". However, any method for detecting forgery of identity documents should efficiently apply for individual suspicious symbol recognition as the negative impact of missed fakes is tremendous compared to the inconveniences of false alarms. Besides in both [6,8] a predefined set of typefaces is considered, while our method should classify fonts not represented in the training set.

PROBLEM STATEMENT AND ITS SPECIAL TRAITS
Let be the set of all existing fonts. Let ′ be the font, specified in standards (for example, OCR-B for machinereadable zones [7]). Let = { } be the set of character images with the following functions defined on it: : → the function of font and the correspondence function : →{0,1}: We need to develop a classifier, implementing the correspondence function . As the whole set is too large or even infinite, we should use ′′, i.e. the set of fonts that are available to create a sample of forged symbols, as an approximation of \ ′. Still, the resulting classifier should work properly on any font from and not only on the subset ′ ∪ ′′.
The images of the Russian national passport [16] used as recognition objects have the following special traits:  Low quality, compression artifacts and low resolution -96 dpi -as they are captured with smartphone cameras in uncontrolled conditions;  Complex background and varying quality of paper and printing devices.

EXPERIMENTS
In our experiments, we use images of digits, i.e. if we consider the recognition alphabet as = { } =1 , in which -alphabet size, thus in our study = 10. The multi-task learning was applied to train a classifier implementing the function : → × {0, 1} ( -type classifier), as it allows to differentiate images by both fonts (i.e. implementing the function from Sec. 3) and characters. We assume that some characters display more variability between the fonts than the other ones. For instance, the images of '4' in Fig.1 (a) and in Fig.1 (b) differ, while the images of '0' are almost similar. The total number of classes is equal to 20 as =10. To evaluate the acquired classifier, we train its analogue implementing function ′ : → {0,1} ( ′-type classifier) which distinguish only the fonts, so its number of classes is equal to 2.
Similarly to published approaches [6,8,9], we consider a sample of images with the font specified in standards (not forged) a negative sample, and a sample with other fonts (forged) -a positive one.
We train CNNs for MRZ and for the Russian national passport ID number, as fonts of these fields are defined in the standards [17,18], while the other fields have no strict limitations [19]. Although the font for MRZ (called OCR-B) that is used in the Russian national passport is publically available, the Russian national passport is a heavily protected document [20] and is declared a level "A" fraud-proof printing product [21]. It means that everything necessary for its production is classified, including the font used for printing ID numbers [21,22]. Examples of digits found in the forged Russian national passports are shown in Fig.1 (c), while their analogues printed with the genuine font are demonstrated in Fig.1 (d).

Evaluation of results
False positive results occurring when the genuine passport is recognized as a fake one are less harmful to most financial or legal procedures, such as loan granting procedures or border control, than a situation, in which a fake ID was mistaken for the genuine one.
To evaluate each trained classifier, we calculate its overall sensitivity and specificity on the three test sets (Section 4.4) and build its confusion matrix. However, the entire confusion matrix for a -type classifier is rather large (400 elements in case of 20 classes). Due to the detrimental impact of false negative errors, it seems reasonable to focus only on the part of the confusion matrix corresponding to the forged fonts. For each -type classifier, we build a modification of the confusion matrix where for each class printed with a forged font we provide:

Network architecture
We suggest using the architecture presented in Fig.2 for -type classifiers. The ′-type classifier has the similar architecture except for the fact that the number of neurons in the last fully connected layer equals 2. This architecture has a rather small number of learnable parameters, about 7500 for a -type network with =10, which makes it usable for smartphone applications. The size of the input image is 15x19 pixels.

Training datasets
In this study, we conduct experiments using digits from two fields of the Russian national passport, applying different approaches to create training datasets. The training set for the classifier for MRZ is synthesized with the method described previously [19]. To create the datasets, we use the fonts available on GoogleFonts [23]. We use OCR-B specified in the MRZ government standards [17,18] to design a negative sample and 17 fonts similar to OCR-B to generate a positive one. Since the font used in the passport ID number cannot be found in public access, it cannot be used for synthetic data generation. Given this, we use augmented natural data as a negative sample. The positive sample was generated from 46 fonts selected on GoogleFonts [23]. The amount of characters in the training datasets is shown in Fig.3. a b Figure 3. Amount of characters in the training sets for the CNN for MRZ (a) and for the passport ID number (b)

Test datasets
To estimate the resulting CNNs, we use a dataset consisting of 3238 various passport images which were not used in the training process. All images were captured with small-scale digital cameras and were manually labeled. In our experiment, we create three distinct image test sets containing individual characters arranged into 3 subtypes:  Digits from the passport ID number (129565 images). This sample is used as the negative one to test the CNN for the passport ID number and as the positive one to test the CNN for MRZ;  Digits from MRZ (68143 images). This sample is used as the negative one to test the CNN for MRZ and as the positive one to test the CNN for passport ID number;  Digits from dates and authority code (109742 images). This sample contains digits acquired from other passport fields (dates of birth, issue dates, authority codes) and is used as the positive one to test CNNs for both MRZ and the ID number.
It should be emphasized that the fonts used to print dates and authority codes are absent in any training set. Moreover, we do not include the MRZ font in the training set of the ID number classifier and vice versa. The amount of characters in the three test sets is demonstrated in Fig.4.

Classifiers for passport ID number
First, we train two classifiers for the passport ID number: a -type classifier and a ′-type classifier. Table 1 provides the results demonstrated by these CNNs.
In the study, we do not take into account errors of character classification for the -type classifie, i.e. an answer is considered to be True Positive if it belongs to type 1 or 2 from the list in Sec. 4.3. Obviously, the -type classifier surpasses the ′-type one in both sensitivity and specificity (Table 1). Table 1. Results of C-type and C′-type classifiers for the passport ID number. TP -character printed with the forged font recognized as such, TN -character printed with the genuine font recognized as such, FP -character printed with the genuine font recognized as a forged one, FN -character printed with the forged font recognized as a genuine one Then, we demonstrate the modification of the confusion matrix described in Sec.4.3 for the forged fonts (Table  2). Here, we denote all the classes by '\_w' to emphasize that we consider only the images with forged ("wrong") fonts. Also, all types of the results are labeled according to the list from Sec.4.3. Our results in Table 2 indicate that error rates of font classification for certain characters are above average. Excluding the symbols with the highest rates of font misclassification, we obtain the sensitivity of the classifier equal to 93.08%. Assuming them being forged, we get the sensitivity reaching 95.12%. In addition, the results provided show that the average error rate of character recognition is 0.88% (maximum is 3.44% for the character '7'). Therefore, we conclude that the proposed CNN can be used for detecting forged fonts, especially if combined with a recognition CNN.

Classifiers for MRZ
The results of training the -type and the ′-type classifiers for MRZ are provided in Table 3. As well as in Table 1, in Table 3 we do not consider errors of character classification. Unlike the experiment with the ID numbers, the ′-type network demonstrates higher sensitivity than the -type one for the forged fonts in MRZ. Yet, it comes with a significant decrease in specificity, as the ′-type network classifies more than a half of genuine symbols as forged. The -type classifier demonstrates much higher specificity, however, losing 2.09% (86.28% against 88.37%) in sensitivity. Table 3. Results of C-type and C′-type classifiers for MRZ. TP -character printed with a forged font recognized as such, TNcharacter printed with the genuine font recognized as such, FP -character printed with the genuine font recognized as a forged one, FN -character printed with the forged font recognized as a genuine one To analyze the errors of -type classifier, we compile Table 4 which is similar to Table 2. Similarly to the measurements for the passport ID number, the results in Table 4 point out that some of the characters in the specified font do not have enough distinguishable features and resemble symbols from the positive sample, especially the symbol '0'. Moreover, if we do not take this character into consideration, the sensitivity reaches 92.56% which outperforms the ′-type classifier. If we always consider '0' being forged, the sensitivity is 94.06%. Following these results, we consider the acquired classifier appropriate for the verification of fonts.  0_w  1_w  2_w  3_w  4_w  5_w  6_w  7_w  8_w  9_w   Result  type   1  29474 37432 24025 13333 13683 17581 14898 14003 16523 21096   2  125  646  1241  320  380  255  861  210  183  216   3  18475  1216  2101  2361  4432  185  305  846  195  601   4  127  188  264  250  229  112  381  248 82 224

CONCLUSION AND FURTHER WORK
In this paper, we have proposed a method based on optical font recognition to analyze the consistency between the font used in a document and the one specified in standards. We consider the cases, in which all the fonts necessary for classifier training are publically available and a sample of the font specified in standards is obtained from images of real documents. The test sets consist of images of digits from different fields of Russian national passports. All test images were captured with small-scale digital cameras. The most important trait of the suggested classifier is its generalization ability, confirmed by the fact that the fonts from the test sets used as forged ones were not present in the training set. We use multi-task learning to train a CNN (called a -type classifier) that recognizes both the character class and the authenticity of the font. Also, we train another classifier that recognizes only the font (called ′-type classifier) to compare these two approaches. The obtained results indicate that:  The classifiers that are distinguishing classes by both characters and fonts ( -type) demonstrate higher sensitivity and specificity than the ones trained only to predict the font class ( ′-type);  The suggested -type classifiers can be used for detection of the forged fonts, but different characters should be analyzed with different weights as some of them do not have enough features. We propose that, when embedded into a document recognition system, the suggested classifiers should be rather used as a part of an authenticity control system than individually.
We expect that the combination of a standard character classifier and the suggested classifier should be efficient on character sequences (e.g. the Russian passport ID number consists of 10 digits). For instance, such a combination can operate as follows: 1. Recognize all symbols with the standard classifier and set weights according to the recognition results; 2. Recognize each symbol with the suggested classifier and, if an unsuitable font is detected or the result of symbol recognition differs from the result of step 1, consider the symbol to be printed with an inadequate font; 3. Evaluate the symbol sequence according to the results of step 2 and the weights from step 1 to make a verdict about the field authenticity.
For future work, we plan to study the applicability of the suggested method for different identity documents with specific fonts, such as French ID cards or British driving licenses. Besides, we will explore the possibility of training ′-type classifiers to a sufficient sensitivity and specificity. This type of classifiers does not have limitations on the number of character classes as the number of neurons in its last layer is always equal to 2, unlike the -type.