{'fraction_non_alphanumeric': 0.028590762984266622, 'fraction_numerical': 0.019116900693622062, 'mean_word_length': 4.740470988103909, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Primary Open-Angle Glaucoma Diagnosis from Optic Disc Photographs Using a Siamese Network

Purpose Primary open-angle glaucoma (POAG) is one of the leading causes of irreversible blindness in the United States and worldwide. Although deep learning methods have been proposed to diagnose POAG, these methods all used a single image as input. Contrastingly, glaucoma specialists typically compare the follow-up image with the baseline image to diagnose incident glaucoma. To simulate this process, we proposed a Siamese neural network, POAGNet, to detect POAG from optic disc photographs. Design The POAGNet, an algorithm for glaucoma diagnosis, is developed using optic disc photographs. Participants The POAGNet was trained and evaluated on 2 data sets: (1) 37 339 optic disc photographs from 1636 Ocular Hypertension Treatment Study (OHTS) participants and (2) 3684 optic disc photographs from the Sequential fundus Images for Glaucoma (SIG) data set. Gold standard labels were obtained using reading center grades. Methods We proposed a Siamese network model, POAGNet, to simulate the clinical process of identifying POAG from optic disc photographs. The POAGNet consists of 2 side outputs for deep supervision and uses convolution to measure the similarity between 2 networks. Main Outcome Measures The main outcome measures are the area under the receiver operating characteristic curve, accuracy, sensitivity, and specificity. Results In POAG diagnosis, extensive experiments show that POAGNet performed better than the best state-of-the-art model on the OHTS test set (area under the curve [AUC] 0.9587 versus 0.8750). It also outperformed the baseline models on the SIG test set (AUC 0.7518 versus 0.6434). To assess the transferability of POAGNet, we also validated the impact of cross-data set variability on our model. The model trained on OHTS achieved an AUC of 0.7490 on SIG, comparable to the previous model trained on the same data set. When using the combination of SIG and OHTS for training, our model achieved superior AUC to the single-data model (AUC 0.8165 versus 0.7518). These demonstrate the relative generalizability of POAGNet. Conclusions By simulating the clinical grading process, POAGNet demonstrated high accuracy in POAG diagnosis. These results highlight the potential of deep learning to assist and enhance clinical POAG diagnosis. The POAGNet is publicly available on https://github.com/bionlplab/poagnet.

POAG classification after extracting the handcrafted feature from vertical cup-to-disc ratio. Acharya et al 9 detected POAG using support vector machines and Naïve Bayesian classifier based on the texture features and higher-order spectral features. Dua et al 10 also used support vector machines and Naïve Bayesian to classify POAG based on the wavelet-based energy features. Issac et al 11 adopted an adaptive threshold-based image processing method for POAG classification. However, the segmentation accuracy of optic disc and optic cup greatly impacts these methods. In addition, these methods only consider handcrafted features, and thus are hard to learn to generalize.
Recently, deep learning methods have demonstrated promising results in biology and medicine. 12e22 In the ophthalmology domain, several methods have been proposed to detect POAG at its early stage. 23e29 However, these approaches all used a single image as input. However, in clinical practice, glaucoma specialists often compare the follow-up image with the baseline image to trace the relevant features and assess for glaucomatous change (Fig 1). It is worth noting that the baseline image must be non-POAG. To simulate this process, we propose a Siamese neural network model, POAGNet, in this work. The Siamese neural network uses the same weights in a twin network while working on 2 different images to compute comparable output. 30 In our case, the output is the baseline image against which the other output is compared. To the best of our knowledge, it is the first time in the ophthalmology domain that 2 optic disc images have been utilized and compared for automated glaucoma detection via Siamese networks.
Unlike previous Siamese work, POAGNet used a convolution operation to study the feature difference between 2 outputs. Compared to the traditional absolute distance, our inner neural network approximates the difference between the 2 inputs more precisely by updating the parameter of convolution during the training. In addition, the POAGNet consists of side output 13 to ease the vanishing gradient problems in training and force the hidden layers to favor discriminative features.
Our study also aimed to evaluate the model's generalization capacity for different data sets not employed in the training process. To this end, we assessed POAGNet on 2 large-scale, independent data sets, the Ocular Hypertension Treatment Study cohort (OHTS) 26 and Sequential fundus Images for Glaucoma data set (SIG), 31 with > 35 000 optic disc images in total.
To the authors' best knowledge, this work is innovative in leveraging the Siamese neural network to compare the differences between 2 optic disc photographs. Thus, our model closely matches the clinical decision-making process, which allows a glaucoma specialist to inspect the result rather than being presented with a "black-box" approach. In addition, the proposed model was validated on 2 large-scale, multi-institutional benchmarks and achieved superior results against several competitive baselines. Therefore, our model is robust and will likely be generalizable to new data. Finally, we make codes, models, and preprocessed data publicly available to catalyze future works that seek to develop deep learning models for POAG detection.

Data Acquisition
In this study, we include 2 independent data sets (Table 1). For the OHTS data set, the number of the non-POAGenon-POAG pairs used from eyes converted to POAG is 29 339. The number of the non-POAGenon-POAG pairs and non-POAGePOAG pairs used from eyes that converted to POAG are 2443 and 2285, respectively. For the SIG data set, these 3 numbers are 3015, 111, and 153, respectively. These 2 databases are large-scale, cross-sectional, and population-based studies. In this study, all eligible subjects are non-POAG at baseline.
OHTS. The first data set is obtained from the OHTS. The OHTS is one of the largest longitudinal clinical trials on POAG (1636 participants and 37 399 images) from 22 centers in the United States. Human subjects were included in this study. The study protocol was approved by the institutional review board at each clinical center and Weill Cornell Medicine. 32 All research adhered to the tenets of the Declaration of Helsinki. All participants provided informed consent. All risk factors were measured at baseline before the onset of the disease and collected for approximately 16 years.
The participants in this data set were selected according to eligibility and exclusion criteria. 33 Briefly, the eligibility criteria include intraocular pressure (between 24 mm Hg and 32 mm Hg in one eye and between 21 mm Hg and 32 mm Hg in the fellow eye) and age (between 40 and 80 years). The visual field (VF) tests were interpreted by the Visual Field Reading Center, and the optic discs at clinical examination and stereoscopic photographs were interpreted by the Optic Disc Reading Center. Exclusion criteria included previous intraocular surgery, visual acuity worse than 20/40 in either eye, and diseases that may cause optic disc deterioration and VF loss (such as diabetic retinopathy). The gold standard POAG labels were graded at the Optic Disc Reading Center. In brief, 2 masked certified readers were arranged to independently detect the optic disc deterioration. If there was a disagreement between 2 readers, a senior reader reviewed it in a masked fashion. The POAG diagnosis in a quality control sample of 86 eyes (50 normal eyes and 36 with progression) showed test-retest agreement at k ¼ 0.70 (95% confidence interval, 0.55e0.85). More details of the reading center workflow have been described by Gorden et al. 32 SIG Database. The second data set is obtained from the SIG data set (https://github.com/XiaofeiWang2018/DeepGF). The study protocol was approved by the institutional review board at each clinical center and Beihang University. All research adhered to the tenets of the Declaration of Helsinki. All participants provided informed consent. The SIG contains 3837 optic disc images, of which 153 (3.99%) have POAG. In the SIG data set, all optic disc images are annotated with binary labels of glaucoma, that is, positive or negative glaucoma. The samples are labeled glaucomatous when they satisfy any of the 3 criteria, that is, retinal nerve fiber layer defect, rim loss, and optic disc hemorrhage. 31

Model Development
Overall Architecture. The POAGNet comprises 2 convolutional blocks that share the weight and are followed by 7 layers (Fig 2). The prediction block in Figure 2 shows the detail of the 7 layers. In the beginning, 2 optic disc images, x 1 and x 2, are passed through the convolutional neural network, DenseNet-201, 34 respectively. We used the output of last (F d4 and F d4n ) and second last Dense Blocks (F d3 and F d3n ). For each output, we concatenated two outputs, followed by using 1 Â 1 convolution, a batch normalization, 35 and rectified linear units. 36 The advantage of a convolution is that, instead of vectors, we can compute the similarity between 2 feature maps at all kernels. Therefore, the output is not a single value but another feature map with spatial support. In the end, a global average pooling 37 and a fully connected layer with sigmoid activation are attached.
The POAGNet connects the side-output layer to the last 2 prediction blocks. The side-output layer generates the inherent scales of discriminative features and relieves the vanishing gradient problems in training. Finally, all side-output layers are averaged to generate a POAG prediction.
Loss Function. In this study, we use binary cross-entropy as the loss function in the POAGNet. In addition, to overcome the severe class imbalance for the POAG classification, we apply the weighted cross-entropy, 38 a commonly used loss function in classification. The adopted weighted cross-entropy was as follows: where N is the number of training examples and b is the balancing factor between positive and negative samples. Here, we used the inversely proportional to POAG frequency in the training data.
The overall loss function is the average of the losses associated with a prediction from the last 2 blocks: Image Augmentation. In this study, the following augmentation techniques were applied on the fly during training: (1) random rotation between 0 and 10 , (2) random translation: an image was randomly translated along the x-and y-axes by distances ranging from 0% to 10% of width or height of the image, and (3) random flipping. These augmentation operations increase the diversity of the data set.

Evaluation Metrics
Our experiments report accuracy, sensitivity (recall), and specificity. In addition, we report the area under the curve (AUC) receiver operating characteristics curve. A receiver operating characteristics curve plots true positive rate (also called sensitivity) versus false-positive rate at different classification thresholds.

Experimental Settings
We first used a single image as input without manual cropping in the POAG detection task to fine-tune a DenseNet-201, which has been pretrained on ImageNet. Then, we used this DenseNet-201 to initialize the subnets (DenseNet-201) in the POAGNet and finetuned the entire network in an end-to-end manner. Therefore, the loss is propagated back to the individual neural networks, creating better feature representations for each training iteration.
All images are resized to 224 Â 224 Â 3 as input of the proposed model. The models were implemented by Keras with a backend of TensorFlow. The proposed network was optimized using the Adam optimizer method. 39 The learning rate is 5 Â 10 À5 . a is 0.8. The experiments were performed on Intel Core i9-9960 X 16 cores processor and NVIDIA Quadro RTX 6000 GPU.
For the OHTS data set, we split the entire data set randomly at the patient level. We took 1 group (20% of total subjects) as the hold-out test set and the remaining as the training set. For the SIG data set, we used the official training, development, and testing split in this study. In addition, we performed cross-data set bias and evaluation: (1) model trained on OHTS and tested on SIG; (2) model trained on SIG and tested on OHTS; (3) model trained on SIG and OHTS jointly and tested on SIG, OHTS, and their combination, respectively.
b y n ðx n ; q s Þ þ ð1 À bÞð1 À y n Þlog 1 À b y n ðx n ; q s Þ i

Results
We compared our method with 6 models on POAG diagnosis on the OHTS data set, including the DenseNet-201 34 with a single image as input, EfficientNetB0 40 with a single image as input, MobileV2 41 with a single image as input that was used in Thakur et al, 26 ResNet-50 with a single image as input that was used in Fan et al, 42 the traditional Siamese network with absolute distance, and POAGNet using the last DenseNet Block (POAGNet w/o side output). DenseNet-201, EfficientNetB0, Mobile-NetV2, and ResNet-50 were pretrained on ImageNet and we modified the fully connected layers so that they could meet our binary classification requirement. We fined-tuned the entire network in an end-to-end manner.

POAG Diagnosis on the OHTS Data Set
We first trained and validated the models on the OHTS data set. Table 2 shows the performance comparison. Our model achieved the best results, with an accuracy of 0.9283, a sensitivity of 0.7469, a specificity of 0.9421, and an AUC of 0.9587. The performance of POAGNet was then compared with that of the models with a single image as input. The performance of POAGNet was superior to that of the best single model, DenseNet-201, with 9.63% higher accuracy, 1.78% higher sensitivity, 1.28% higher specificity, and 8.37% higher AUC. In addition, the performance of POAG was compared with 2 variations, 1 without side output and 1 with absolute distance. The POAGNet obtained an AUC of 0.9587, resulting in an improvement of 5.1% over the POAGNet with the absolute distance (row 4) and 3.5% over the POAGNet w/o side output (row 5). To check how many side outputs are optimal, we also performed the experiments where the POAGNet used the side output of the last 3 Basic Blocks as well as all Basic Blocks. It turns out that the proposed structure, which used the side output of the last 2 Basic Blocks, achieved the best results (Table S1). We also tried the contrast loss instead of binary cross-entropy loss, and the results decreased (Table S2).

POAG Diagnosis on the SIG Data Set
We then trained and validated POAGNet on the SIG data set. Table 3 compares the performance of POAGNet on the SIG data set. The POAGNet obtained the best results, with an accuracy of 0.9176, a sensitivity of 0.1786, a specificity

Cross-Data Set Bias and Evaluation
In separate experiments, to assess the generalizability and transferability of POAGNet, we compared the performance of models trained on OHTS, SIG, and their combination (OHTS þ SIG) (Fig 3). For accuracy and specificity (Fig 3A, C), our models achieved comparable results in different training and testing scenarios, indicating that our model can generalize and transfer across different data sets.
For sensitivity and AUC (Fig 3B, D), POAGNet achieved comparable or better performance when trained on OHTS or OHTS þ SIG but weak performance when trained on SIG only. When POAGNet is trained on SIG only, the training size is relatively small, which makes it difficult to fully train the model. We will explain observations in more detail in the "Discussion" section.
We also sampled the OHTS testing data set, making the number of instances in the OHTS testing data set the same as those in the SIG testing data set. The AUC is 0.8797 (Table S3).

Discussion
This study proposed a new end-to-end deep learning network that simulates the clinical process of automatic POAG detection from optic disc photographs. Two data sets were used to evaluate the proposed model. The results demonstrated that the proposed network was superior to the state-ofthe-art model. The proposed model has potential for eye services in the future after extensive validation across multiple and diverse image data sets. Unlike the previous networks that use a single image as input, our proposed model simulates the clinical process by comparing the differences between 2 input images (baseline and follow-up images).
The POAGNet achieved superior performance to the state-of-the-art methods for 3 reasons. First, the proposed network used the Siamese neural network and 2 images (baseline and follow-up) as input. This architecture simulates the clinical process of identifying POAG from optic disc photographs by comparing the follow-up image with the baseline image. Tables 2 and 3 show that the Siamese neural networks were superior to DenseNet-201, suggesting that the Siamese network can leverage the differences between 2 images for a more accurate POAG diagnosis. Second, POAGNet used a convolution operation to measure the difference between 2 outputs. The advantage of the convolution operation is that it computes the similarity on all kernels, and the parameters in the convolution can be updated during the training. The output of this operation is not a single score but rather a feature map with spatial support. As shown in Tables 2 and 3, POAGNet w/o side output achieved a better result than the POAGNet with absolute distance, indicating that utilizing convolution to measure the similarity between 2 outputs could boost the performance. Third, the POAGNet used side output 13 to relieve the vanishing gradient problems in training and encourage the hidden layers to favor discriminative features. By comparing rows 5 and 6 in Tables 2 and 3, POAGNet can leverage multiscale information to boost the model's performance.
We also analyzed the causes of misclassification. We found that the major reason for misclassification is due to the optic disc unobvious discrepancy in the early stage. Fig S1 shows the positive predictive value, sensitivity, specificity, accuracy, and AUC in each time interval. Specificity, accuracy, and AUC are almost identical during these time intervals. Additionally, there are only 1 and 4 POAG cases for the first 2 time intervals, which makes the sensitivity relatively high. The sensitivity is also relatively stable for the rest time intervals. On the other hand, the positive predictive value has a continuing upward trend, indicating that nonPOAG-POAG pairs are easier to detect at the late stage than at the early stage. Second, we counted the number of misclassified patients in each relative year after the POAG onset year. Table 4 shows that more patients are misclassified when their POAG onset year is closer to the baseline visit. We also studied if the proposed model can focus on the important region to detect POAG. Because class activation maps are usually applied for the model with a single input, we used another method to reflect the regions that were more important for POAG diagnosis. Specifically, we used a 5 Â 5 window size with a step size of 1 to mask the image and obtain the prediction probability for each window. We obtained the probability of each pixel by averaging all the probabilities of the windows it belongs to. Therefore, we can get the saliency map by drawing the probability of each pixel, which can reflect the important region for POAG detection. Four examples in Figure 4 demonstrate that the proposed model focused on the region between the optic disc and cup to detect POAG.
We conducted further experiments to evaluate the POAGNet performance on external validation data sets. Comparing the models trained on OHTS, SIG, and their combination, AUCs are similar on the SIG test set. This observation demonstrates the relative generalizability of POAGNet. On the other hand, we observed that the AUC on OHTS drops from 0.9587 (trained on OHTS) to 0.7410 (trained on SIG). One potential reason is that the ophthalmologists carefully selected the images in SIG, and all glaucomatous images are due to the structural glaucomatous optic nerve abnormalities. 31 In contrast, some optic disc photographs in OHTS were graded as POAG due to the change of VF without glaucomatous disc changes over time. Therefore, detecting POAG due to VF defects remains challenging. Finally, we compared our work with 2 previous works based on the OHTS data set. We found 2 experimental differences when comparing our work with the prior work of Thakur et al. 26 First, Thakur et al 26 discarded 24% of the optic disc photographs with poor image quality, while we used the whole OHTS data set. Second, Thakur et al 26 manually cropped the images in the data preprocessing stage, but we did not. It is possible that the use of these techniques might have improved the accuracy of the model. However, we deliberately avoided this extensive preprocessing to minimize the labor required and make our model as generalizable as possible. To make a fair comparison, we applied the same method in Thakur et al 26 to our data set (MobileV2 in Table 2). The AUC was 0.8124, lower than POAGNet under the same setting. In addition, our proposed model achieved an AUC of   42 (0.88). Because we did not use the same training and testing data split and preprocessing methods, the results are not strictly comparable. We also used these 3 models to test our proposed model, and the AUCs are 0.9378, 0.9286, and 0.9587, respectively.

Limitations and Future Work
One limitation of our proposed model comes from the data imbalance. Only 6.2% and 3.99% of all images had POAG on OHTS and SIG, respectively. The low portion of POAG images may result in relatively lower sensitivity. We plan to incorporate data sets from different countries and populations to improve the model in the future.
Another potential limitation is that, as previously discussed, it remains challenging to only detect POAG due to VF defects. This is mainly because the VF defects lack the obvious structural sign of glaucomatous optic neuropathy. It would be interesting to study the difference between the results derived from VF defects and glaucomatous optic nerve separately and incorporate VF into the deep learning models for a multimodal study.
In this work, we used baseline images and "follow-up images" as input to obtain substantial differences between images for POAG detection. The longitudinal images between the baseline image and the "follow-up image" may also play an important role in tracing the temporal patterns of POAG progression. Thus, we recommend using sequential models such as long short-term memory or transformer for POAG detection in the future.
In conclusion, this study proposed a new end-to-end deep learning network that simulates the process for automatic POAG detection from optic disc photographs. Two data sets were used to evaluate the proposed model. The results demonstrated that the proposed network performs well on POAG diagnosis. The cross-data set validation further proves the generalizability and robustness of our method. To maximize this study's transparency and reproducibility and provide a benchmark for further refinement and development of the algorithm, we make the deep learning model, training codes, and data partition publicly available (https:// github.com/bionlplab/poagnet).