{'fraction_non_alphanumeric': 0.03160599900272333, 'fraction_numerical': 0.011967320010739902, 'mean_word_length': 5.050591784636807, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Projective Transformation Rectification for Camera-captured Chest X-ray Photograph Interpretation with Synthetic Data

Automatic interpretation on smartphone-captured chest X-ray (CXR) photographs is challenging due to the geometric distortion (projective transformation) caused by the non-ideal camera position. In this paper, we proposed an innovative deep learning-based P rojective T ransformation R ectification N etwork (PTRN) to automatically rectify such distortions by predicting the projective transformation matrix. PTRN is trained on synthetic data to avoid the expensive collection of natural data. Therefore, we proposed an innovative synthetic data framework that accounts for the visual attributes of natural photographs including screen, background, illuminations, and visual artifacts, and generate synthetic CXR photographs and projective transformation matrices as the ground-truth labels for training PTRN. Finally, smartphone-captured CXR photographs are automatically rectified by trained PTRN and interpreted by a classifier trained on high-quality digital CXRs to produce final interpretation results. In the CheXphoto CXR photograph interpretation competition released by the Stanford University Machine Learning Group, our approach achieves a huge performance improvement and won first place ( ours 0.850 , second-best 0.762, in AUC). A deeper analysis demonstrates that the use of PTRN successfully achieves the performance on CXR photographs to the same level as on digital CXRs, indicating PTRN can eliminate all negative impacts of projective transformation to the interpretation performance. Additionally, there are many real-world scenarios where distorted photographs have to be used for image classification, our PTRN can be used to solve those similar problems due to its generality design.


Introduction
Chest X-ray (CXR) is one of the most ubiquitous medical imaging techniques for chest disease diagnosis. CXR interpretation requires certified radiologists. However, due to the lack of resources, recently some CXRs cannot be formally reviewed by radiologists in hospitals [1].
For instance, over 300 thousand patients in England wait more than 30 days for the reports of their X-rays [2].
Recently, researchers have successfully developed various deep convolutional neural networks (CNNs) to perform automatic interpretation on digital CXRs [3][4] [5], which can outperform the average level of radiologists in certain disease detection [3]. For example, in 2021, Yuan et al. [5] trained a deep learning ensemble that can automatically interpret digital CXRs into 5 pathologies with a mean AUC of 0.93 and has beaten 2.8/3 radiologists on the CheXpert dataset [6].
Unfortunately, in many developing countries and regions, traditional X-ray films are still widely used [7][8] [9]. The CXR films are hard copies that cannot be directly interpreted by computer-based AI models. A quick and convenient solution to perform automatic interpretation is using a smartphone camera to take a photograph of a CXR film (referred to as CXR film photograph) or a digital CXR on a monitor screen (referred to as CXR monitor photograph). Benefitting from the low prices and popularity of smartphones, CXR photograph interpretation can be a cheap and effective solution for countries and regions which have very limited medical resources. Moreover, the use of smartphones can facilitate both real-time and store-and-forward medical consultation to provide remote medical care and remote diagnosis [10] [11]. Especially, the sudden outbreak of COVID-19 in 2020 has promoted the practical applications of telemedicine and zero-contact diagnosis. Furthermore, it can help patients who have privacy concerns that not willing to go to hospitals for radiology reports.
However, automatic interpretation on smartphone-captured CXR photographs is a very challenging task due to the visual attributes such as: (1) the CXRs in photographs could be geometrically distorted due to non-ideal camera position [12], as shown in Figure 1. The distortion is known as projective transformation, as the CXRs appear on flats like monitor screens or X-ray viewers. The distortions are widely varying due to different camera positions; (2) the illuminations are also varied (daylight, indoor lighting, the backlight of the monitor, etc.); (3) other visual artifacts and noises, such as camera out-focus, image noise, and moiré pattern, could appear. Therefore, the performance of current existing interpretation algorithms with well-trained CNN models on these remake CXR photographs experiences a significant decline, as those algorithms were trained on high-resolution digital CXRs [13][14] [15].
Among these visual attributes, Chong et al. [14] have experimentally verified that one major reason for such performance decline is the geometric distortion (projective transformation) of CXR photographs. Several studies also have experimentally demonstrated that the geometric distortion causes significant performance declines on several image recognition tasks such as image classification [16] and text recognition [17]. Therefore, one obvious solution is to rectify the projective transformation of CXR photographs.
In the literature, deep learning-based methods have been proposed to automatically rectify the geometric distortion for different camera-captured photographs. Digital CXR CXR Photograph Figure 1: The CXR in a photograph experiences geometric distortion (projective transformation) due to non-ideal camera position.
GAN-STAM was proposed by Chong et al. [14] that utilizes GAN [18]  In this paper, we proposed an innovative method for CXR photographs rectification, called Projective Transformation Rectification Network (PTRN). PTRN predicts the projective transformation matrices, which has not been seen in other researchers' work, to automatically rectify the projective transformation of CXR photographs precisely. Additionally, to tackle the problem of the lack of training data identified in the paper [14], we proposed an innovative framework to generate synthetic CXR photographs and projective transformation matrices as the ground truth labels for training PTRN.
The primary contributions of this work are summarized: 1. To the best of our knowledge, our innovative PTRN is the first rectification method that predicts the projective transformation matrix to rectify the projective transformation caused by non-ideal camera positions. PTRN has a simple architecture trained on synthetic data. Additionally, the Intersection over Union (IoU), which is essentially used to quantify the percent overlap for image segmentation but never be used for the evaluation of rectification performance, is firstly proposed to quantitively evaluate the rectification performance on CXR photographs.

CXR Photograph Interpretation
The interpretation performance of various well-trained CNN models experiences a significant decline on CXR photographs [13][14] [15].
In 2020, Le et al. [26] proposed a method to crop the CXR region in each CXR photograph using YOLO-v3 [27] to remove the unwanted background. However, the classification performance is extremely low and unacceptable (AUC 0.684).

Geometric Distortion Rectification for Camera-captured Photographs
Traditional non-AI based methods to solve the geometric distortion rectification problem for  [20] achieves rectification in the way similar to DocUNet but further exploits 3D shape representation to improve the rectification performance. These methods are trained on synthetic data like Doc3D [20]. However, these methods are particularly designed for curved and folded documents, which are incompatible with photographs that experienced projective transformation.
Another example is Automatic License Plate Recognition (ALPR) which requires deep learning-based methods, as the appearances of plate photographs captured by CCTV cameras vary, including different illuminations, weather, etc. Silva and Jung [23] proposed an object detection network called IWPOD-NET which predicts both the object probability and the 6parameters of the affine transformation to locate a distorted object. Björklund et al. [22] proposed an object detection network trained with synthetic data to predict both the object probability and the corners of the plate. However, these networks cannot perform precise geometric distortion rectification on camera-captured photographs like CXR photographs, since (1) the purposes of these networks are image recognition, instead of image rectification; and (2) they used affine transformation to reduce the distortion, which cannot precisely rectify the distortion (projective transformation) caused by non-ideal camera position in theory [28].
Therefore, these methods are incompatible with photographs that experienced projective transformation.
Besides, deep learning algorithms are also proposed for rectifying other camera-captured photographs such as barcodes [37] and text [17]. The method for barcodes involves the appearances of barcodes (vertical lines), while the geometric distortions of texts include text line curvature which is very different from CXR photographs. Therefore, these rectification methods are also incompatible with photographs that experience projective transformation.
Therefore, we proposed an innovative PTRN which is the first method that predicts the projective transformation matrix of a photograph to rectify the projective transformation caused by non-ideal camera position.

Synthetic Data on Camera-captured Photograph Geometric Distortion Rectification and Object Detection
Synthetic data is one popular method to solve the problem of the lack of data for training deep learning models, as it avoids collecting expensive natural data and annotations. Synthetic data has also been used for object detection [ [38], a text usually appears in well-defined regions like a sign or a flat wall. Therefore, the generation step includes geometric estimation and segmentation to ensure the texts are properly placed in the correct positions.
Moreover, the ground truth labels of synthetic data for different tasks are very different. For object detection, bounding boxes are used to locate the objects. For image dewarping, the ground truth labels are grid meshes.
Our proposed framework for synthetic data generation has a couple of novelty in comparison to previous synthetic data frameworks. We proposed the first framework for CXR photograph generation. The design of the framework is under consideration of the visual attributes of CXR photographs. Furthermore, our framework is the first that uses the projective transformation matrix as the ground truth to represent the distortion caused by the non-ideal camera position.
Unlike object detection tasks that use bounding boxes that only locate the object, the projective transformation parameters can be used to locate and rectify the image, as the recognition performance could achieve higher after rectification [14] [17]. Which is innovative and effective.

Methods
To tackle the identified limitations of GAN-STAM [14] which is a previous work of CXR

Problem Formulation
This section formulates the problem of geometric distortion rectification for CXR photographs.
The designs of PTRN and the synthetic data framework are based on the following formulations.
A CXR photograph ℎ can be treated as a composition of a distorted CXR image and a background image (Figure 2(a)). We borrow a notation ⊕ from [41] to represent the composition of two images. ℎ can be formulated as: where is a CXR image warped by projective transformation with a 3-by-3 transform matrix : = ( ) . A matrix has 8 parameters that represent a projective A CXR photograph ℎ can be rectified using inverse projective transformation if the matrix is known (Figure 2 (b)): where ′ denotes the rectified ℎ .
The design of PTRN (Section 3.2) is based on Eq. (2). A CXR photograph ℎ can be automatically rectified by predicting the matrix . Therefore, a training sample ൫ ℎ , ൯ for PTRN consists of a CXR photograph ℎ and the ground truth matrix .
The design of the synthetic data framework (Section 3.3) is based on Eq. (1). A training sample ൫ ℎ , ൯ can be synthesized using a random CXR image , a random background and a randomly generated matrix .

Projective Transformation Rectification Network
We proposed the Projective Transformation Rectification Network (PTRN) for rectifying the projective transformation of CXR photographs. PTRN predicts the projective transformation matrix of a CXR photograph ℎ . PTRN is similar to the localization net of STN [42] which both predict the transformation parameters. However, PTRN is trained directly using synthetic data while STN is trained using the gradients propagated from back layers.

CXR Region Prediction
A matrix can be converted to a quadrilateral that represents a CXR region. The 4 vertices ( 1 ) of the quadrilateral represented by are: where (  The quadrilateral can also be converted back to a matrix using Eq. (3). This relation is used in the evaluation of the rectification performance of PTRN. For instance, a predicted matrix ̂ can be converted to a predicted CXR region.

Training
PTRN is trained with synthetic training samples {( ℎ ( ) , ( ) )} in an end-to-end manner. As PTRN predicts the parameters by regression, mean squared error (MSE) loss is used to calculate the loss between the model prediction and the ground truth. MSE loss is defined as where denotes the 8 parameters of the ground truth : =
The ground-truth label is the 4 vertices instead of matrix since marking the 4 vertices is the simplest way to annotate ground-truth labels in practice.
We proposed using IoU between the ground truth CXR region and the predicted CXR region for each sample to measure the rectification performance, as the predicted region should be as overlapped as the ground truth region. The predicted region is calculated from the predicted matrix ̂ using Eq. (3) and the ground truth region is given in the validation/test samples. The IoU calculation of a sample is demonstrated in Figure 4.

Training Data Synthesis
Our proposed PTRN is deep learning-based, which requires a large amount of training data.
Unfortunately, there are no suitable public CXR datasets. Collection and annotation of natural data samples that include CXR photographs and ground-truth labels are also expensive.
Therefore, we proposed a framework for the generation of synthetic training samples. The generation of one synthetic training sample consists of 4 steps, as shown in Figure 5: (1)  Randomness is involved in the framework to increase the diversity of synthetic CXR photographs so that hopefully the trained PTRN can be generalized to photographs captured in various scenarios. An ablation study is conducted in Section 4.6 to verify the framework. The

Vertex of predicted region Vertex of ground-truth region
Predicted Region Ground-truth region + = Eq. (9) framework only employs simple image processing methods such as transformation, composition, and simple augmentation methods. Through experiment, the generation speed is around 102.89 samples/second in a PC with Intel i9-10900K CPU.

Screen Synthesis
In a CXR photograph, the distorted CXR is normally surrounded by a dark area, which is usually a monitor screen, as shown in Figure 6. The appearances of the dark areas look like a part of the CXR. This step is to synthesize such a situation to improve the robustness of PTRN in this situation.
Considering that not all the CXR photographs have such a situation, in experiments, the probability of screen synthesis is set to be 0.3. If the screen is synthesized, the output of this step is a CXR with a synthetic screen , otherwise, the output is identical to the input .
The process of screen synthesis has two steps, as shown in Figure 7.
Step 2: CXR Projective Transformation Step 3: Adding Background Step 4: Illumination, Artifacts and Noises Step   Step 1: is scaled-down and translated, determined by the randomly generated padding sizes , , , (see Figure 7). In the experiments, we set: , , , ~ [0,0.6] . Subsequently, the operation of scaling down and translation is implemented by projective transformation. The transform matrix is calculated by: . Finally, is scaled-down and translated: ( ).
Step 2: ( ) and a plain dark color image are composited to get the output : . In the experiments, the RGB values of the is randomly generated by , , ~ {0, 19}.

CXR Projective Transformation
This step is to simulate the projective transformation caused by the non-ideal camera position.
Firstly, a projective transformation matrix is randomly generated to determine how a CXR is distorted geometrically (i.e., determines the position of the distorted CXR). Then, projective transformation with matrix is applied to the output of step 1 to get a distorted CXR .
The 8 parameters of must be generated under some regulations since an inappropriate set of parameters results in a nonsensical distorted CXR, e.g., the shape is non-convex [45] [28].
Therefore, to avoid such unexpected situations, we split the matrix into a sequential series of actions: scaling, shearing, rotation, perspective warp and translation. Subsequently, the parameters of these actions are generated within an appropriate range (e.g., the parameters of shearing cannot be too large).
The transformation of matrix is split into the 5 actions: The generated matrix is then used to transform the output of step 1 to calculate the distorted CXR .
Step 1 has two types of outputs: and . In the case of , is calculated by . In the case of , since the CXR region in has been transformed by , the transformation by must be inversed to ensure that can correctly represent the CXR region in . The distorted CXR is calculated by

Adding Background Image
In practical situations, the background of a CXR photograph varies. We consider it as a random image . This step is to composite the output of step 2 and a background image. The output is calculated by = ⊕ (9).

Adding Illumination, Visual Artifacts, and Noises
The goal of this step is to make the synthetic CXR photographs more challenging than the natural CXR photographs so that our rectification approach could be more generalized. In the experiments, various data augmentation methods are used to simulate the natural environment such as illumination, visual artifacts and noise, as listed in Table 1.
In the end, ℎ and matrix are composited to be a synthetic training sample ൫ ℎ , ൯. Figure 8 shows some samples of generated ℎ .

Experimental Results and Discussion
All experiments are conducted on Python 3 with TensorFlow 2 framework. The data augmentation methods are implemented by using imgaug library.  (Figure 9 (a)).

CheXphoto
CheXphoto [24] is a public CXR photograph dataset for competition. It consists of 10,507 smartphone-captured CXR photographs from 3,000 patients, which are generated from some digital CXR images in CheXpert. The label for each CXR photograph is inherited from CheXpert. The validation set consists of synthesis photographs and natural photographs. In the experiments conducted in this paper, only natural photographs are used for evaluation. The validation set has two types of natural photographs: (1) A total of 234 CXR monitor photographs (Figure 9(b)). They are the photograph version of the CheXpert validation set.
Each photograph is produced by using a smartphone camera to capture digital CXRs displayed on a monitor. (2) A total of 250 CXR film photographs (Figure 9(c)). Each photograph is produced by photographing CXR films using a smartphone.
In Section 4.4, PTRN is used for the CheXphoto competition. In the competition, the private test CXR photographs were split into two private test sets by type (monitor/film). Therefore, we split the CheXphoto validation set into two sets by the type (monitor/film) for experiments, referred to as the CheXphoto-Monitor validation set and the CheXphoto-Film validation set, respectively. Note that the CheXphoto-Monitor validation set is the photograph version of the CheXpert validation set (digital CXR), we used these two sets to demonstrate the effectiveness of PTRN in Section 4.5.

Implementation Details of PTRN
In the implementation, a DenseNet-201 [44] pretrained on the ImageNet dataset [46] is used as the backbone network. Adam optimizer [47] with a learning rate of 1 × 10 −5 and parameters

CXR Interpretation Pipeline
The pipeline to perform CXR photograph interpretation, as shown in Figure 10, consists of three steps: (1) PTRN predicts the projective transformation matrix of a CXR photograph; (2) the photograph is rectified using the predicted matrix ̂; (3) A classifier trained on high-quality digital CXRs interprets the rectified CXR photograph. This pipeline is used in the experiments shown in Sections 4.4 and 4.5.

CheXphoto Competition
CheXphoto is a competition for X-ray interpretation based on smartphones captured CXR films photographs and CXR monitor photographs, hosted by Stanford and VinBrain [24]. A submitted model is tested to perform multi-label classification on the two private test sets, respectively: for different labels are different. Therefore, we employed binary classifiers to obtain optimal performance for each pathology. The different configurations include different CNN models, learning rates, batch sizes, configurations of label smoothing regularization from [4] for handling uncertainty labels, and data augmentation methods such as adjusting brightness, contrast, and adding Gaussian noise.
Cross entropy loss is used to calculate the loss. The validation dataset is the CheXphoto-Film validation set in which the geometric distortion is perfectly rectified by manual operation.
The results in the CheXphoto competition of the pipeline are reported in Table 2. PTRN outperforms the work in [26] and [14], which are published in the literature. Since the CXRs of these validation/test sets are from different patients, the AUC scores are not comparable across different sets.
By the time of this manuscript submission, our pipeline achieves first place on the leaderboard [25], as shown in Table 3. The first, second, and third places are all our work based on PTRN.
As the competition is released recently, there is no published work on the leaderboard. We compare our pipeline with the fourth place, our approach yields a huge performance gap on CXR film photographs (0.850 vs. 0.762, 0.088 higher in AUC).

Ablation Study on Training Sample Synthesis
The synthetic data framework consists of 4 steps. However, from Eq. (1), a synthetic CXR photograph can be composited by only the distorted CXR and a background. It means in the framework, only step 2 (projective transformation) and 3 (adding background) are necessary.
Step 1 (screen synthesis) and step 4 (adding illumination, visual artifacts and noises) are additional to simulate the visual attributes of CXR photographs. To study the rectification performance impact of steps 1 and 4, we conducted an ablation study by removing step 1 or/and 4.
The results are shown in Table 5. Mean IoU (mIoU) with 95% confidence interval (C.I.) is reported. In the first row, the mIoU of PTRN that is trained with all four steps is reported. It achieves mIoU 0.942/0.892 on the validation/test set respectively. After removing the step(s), the mIoU on the validation set are dropped by approximately 0.02-0.07, and the mIoU on the test set are dropped much larger (approximately 0.07-0.10). It demonstrates the necessity and importance of step 1 and 4. It also verifies that the framework for the generation of synthetic CXR photographs has sufficient variance of the key variables controlling the distorted CXRs.

Qualitative Evaluation of PTRN
In Figure 11, we demonstrate using trained PTRN to automatically rectify six CXR photographs captured in different scenarios. All ix CXR photographs are properly. Especially in the fifth photograph from the left, even if parts of the CXR region are cropped, PTRN can still accurately predict the region and perform rectification. It is because some synthetic samples also have such situations (see Figure 8). These results demonstrate that PTRN is robust to CXR photographs captured in different scenarios. Step CheXphoto  Table 5: Ablation study on the framework for generation of synthetic data (In mean IoU with 95% C.I.).

Conclusion
In this paper, we presented a deep learning-based Projective Transformation Rectification Network (PTRN) that is trained with synthetic data for rectifying the projective transformation of CXR photographs. We also proposed a framework to generate synthetic data. Our pipeline achieves first place on the CheXphoto competition leaderboard, which has a significant improvement over [26] and [14]. This result verifies the performance of our PTRN and the framework for the generation of synthetic data. Additionally, the design of PTRN and the framework for the generation of synthetic data can be applied to other image recognition fields that encounter similar image distortion caused by the imperfect camera position. This work has certain limitations. In Section 4.2, the hyperparameters were not particularly tuned in the training of PTRN since it already produced satisfying rectification performance.
The performance could be further improved by tuning hyperparameters or choosing other CNNs as the backbone. Moreover, further investigation is needed to verify the generalization of PTRN on CXR photographs captured in more scenarios.
We used chest X-ray photographs to demonstrate the effectiveness of our approach, which may also apply in other types of medical imaging such as CT and MRI, or other fields such as Automatic License Plate Recognition. A further investigation is needed to verify.
The above limitations may lead to possible directions to extend or improve this work. The final goal for us is developing a smartphone application for automatic medical imaging interpretation using this work to improve global public health.