{'fraction_non_alphanumeric': 0.036976066860833226, 'fraction_numerical': 0.013106242877041915, 'mean_word_length': 4.904672897196262, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
OXnet: Omni-supervised Thoracic Disease Detection from Chest X-rays

Chest X-ray (CXR) is the most typical diagnostic X-ray examination for screening various thoracic diseases. Automatically localizing lesions from CXR is promising for alleviating radiologists' reading burden. However, CXR datasets are often with massive image-level annotations and scarce lesion-level annotations, and more often, without annotations. Thus far, unifying different supervision granularities to develop thoracic disease detection algorithms has not been comprehensively addressed. In this paper, we present OXnet, the first deep omni-supervised thoracic disease detection network to our best knowledge that uses as much available supervision as possible for CXR diagnosis. We first introduce supervised learning via a one-stage detection model. Then, we inject a global classification head to the detection model and propose dual attention alignment to guide the global gradient to the local detection branch, which enables learning lesion detection from image-level annotations. We also impose intra-class compactness and inter-class separability with global prototype alignment to further enhance the global information learning. Moreover, we leverage a soft focal loss to distill the soft pseudo-labels of unlabeled data generated by a teacher model. Extensive experiments on a large-scale chest X-ray dataset show the proposed OXnet outperforms competitive methods with significant margins. Further, we investigate omni-supervision under various annotation granularities and corroborate OXnet is a promising choice to mitigate the plight of annotation shortage for medical image diagnosis.


Introduction
Modern object detection algorithms often require a large amount of supervision signals. However, annotating abundant medical images for disease detection is infeasible due to the high dependence of expert knowledge and expense of human labor. Consequently, many medical datasets are weakly labeled or, more frequently, unlabeled [22]. This situation especially exists for chest X-rays (CXR), which is the most commonly performed diagnostic X-ray examination. Apart from massive unlabeled data, CXR datasets often have image-level annotations that can be easily obtained by text mining from numerous radiological reports [27,9], while lesion-level annotations (e.g., bounding boxes) scarcely exist [7,28]. Therefore, efficiently leveraging available annotations to develop thoracic disease detection algorithms has significant practical value.
Omni-supervised learning [20] aims to leverage the existing fully-annotated data and other available data (e.g. unlabeled data) as much as possible, which could practically address the mentioned challenge. Distinguished from previous studies that only include extra unlabeled data [20,6,25], we target on utilizing as much available supervision as possible from data of various annotation granularities, i.e., fully-labeled data, weakly-labeled data, and unlabeled data, to develop a unified framework for thoracic disease detection on chest X-rays.
In this paper, we present OXnet, a unified deep framework for omni-supervised chest X-ray disease detection. To this end, we first leverage limited bounding box annotations to train a base detector. To enable learning from weakly labeled data, we introduce a dual attention alignment module that guides gradient from an image-level classification branch to the local lesion detection branch. To further enhance learning from global information, we propose a global prototype alignment module to impose intra-class compactness and inter-class separability. For unlabeled data, we present a soft focal loss to distill the soft pseudo-labels generated by a teacher model. Extensive experiments show that OXnet not only outperforms the baseline detector with a large margin but also achieves better thoracic disease detection performance than other competitive methods. Further, OXnet also show comparable performance to fully-supervised method with fewer fully-labeled data and sufficient weakly-labeled data. In summary, OXnet can effectively utilize all the available supervision signals, demonstrating a promisingly feasible and general solution to real-world applications.

Method
Let D F , D W , and D U denote the fully-labeled data (with bounding box annotations), weakly-labeled data (with image-level annotations), and unlabeled data, respectively. As illustrated in Fig. 4, OXnet correspondingly consists of three main parts: a base detector to learn from D F , a global path to learn from D W , and an unsupervised branch to learn from D U . As the first part is a RetinaNet [15] backbone supervised by the focal loss and bounding box regression loss, we will focus on introducing the latter two parts in the following sections.

Dual Attention Alignment for Weakly-supervised Learning
To enable learning from D W , we first add a multi-label classification global head to the ResNet [5] part of RetinaNet. This global head consists of a 1×1 convolutional layer for channel reduction, a global average pooling layer, and a sigmoid layer. Then, the binary cross entropy loss is used as the objective: where c is the index of total N categories, y c is the image-level label, and p c is the prediction. The 1×1 convolution gets input M and outputs feature map X with N channels, where X is exactly the global attention map, i.e., class activating map [32]. Meanwhile, the local classification head of RetinaNet conducts dense prediction and generates outputs of size Width×Height×Classes×Anchors for each feature pyramid level [14]. These classification maps can be directly used as the local attentions for each class and anchor. Hence, we first take the maximum on the anchor dimension to get the attentions A on number of M pyramid levels. Then, the final local attention for each class is obtained by: Essentially, we observed that the global and local attentions are often mismatched as shown in Fig. 2  covers more precise ROIs by learning from lesion-level annotations. Therefore, we argue that the local attention can be used to weigh the importance of each pixel on X c . We then resize the local attention A c to be the same shape as X c and propose a dual attention alignment (DAA) module as follows: where σ is the sigmoid function, i is the index of the pixels, and denotes element-wise multiplication. Particularly, we let i R c = 1, so the elementwise multiplication constructs a pooling function as in the multiple instance learning literature [8,23]. Then, DAA is used to replace the mentioned global head as illustrated in Fig. 2 (2) and (3). Consequently, the local attention helps rectify the decision-making regions of the global branch, and the gradient from the global branch could thus flow to the local branch. To this stage, the local classification branch not only receives the strong yet limited supervision from D F but also massive weak supervision from D W .

Global Prototype Alignment for Multi-label Metric Learning
With DAA, any global supervision signals could flow to the local branch. To further enhance this learning process, we propose a multi-label metric learning loss that imposes intra-class compactness and inter-class separability to the network. It is worth noting that multi-label metric learning can be hard as the deep features often capture multi-class information [17]. Specially, in our case, we can obtain category-specific features under the guidance of local attention as follows: where M is the feature map before the global 1×1 convolutional layer. Particularly, R c is of shape W × H with W and H being width and height, respectively, i is the index of the feature vectors (M contains #W × H vectors), and is the element-wise multiplication over the dimensions of W and H. Here, the local attention R c highlights the feature vectors related to each category, leading to the aggregated category-specific feature vector F c . For better metric learning regularization, we generate global prototypes [29,31] for each category as follows: where β is 0 if t = 0 otherwise 0.7, and K is the number of data in a batch. Particularly, the prototype is updated with the weighted average of confidences [21,30]. We then align F c with the prototypes as follows: where δ is a scalar representing the inter-class margin.

Soft Focal Loss for Unsupervised Knowledge Distillation
To learn from the unlabeled data, we first obtain a mean teacher model by exponential moving average [24]:θ t+1 = λ ·θ t + (1 − λ) · θ t+1 , where θ and θ are the parameters of RetinaNet and the teacher model, respectively, λ is a decay coefficient, and t represents training step. Then, the student RetinaNet will learn from the teacher model's classification predictions, which are probabilities in range (0, 1). Particularly, the foreground-background imbalance problem [15] should also be taken care of. Therefore, we extend the focal loss to its soft form inspired by [12,26]. Specifically, the original focal loss is as follows: where α is a pre-set weight for balancing foreground and background, q is the local classification prediction from the model (we eliminate the subscript c for simplicity), γ is a scalar controlling (1 − q) γ to assign more weights onto samples which are less-well classified. We obtain the soft focal loss as follows: Here, instead of assigning α according to whether an anchor is positive or negative, we assume its value changes linearly with the value of teacher model's predictionq and modify it to beqα + , where is a constant. Meanwhile, we notice that (1 − q) γ in Eq. (7) depends on how closer q is to its target ground truth. Therefore, we modify the focal weight to be q − q γ to give larger wights to the instances with higher disagreement between student-teacher models.

Unified Training for Deep Omni-supervised Detection
The overall training loss sums up the supervised losses [15], weakly-supervised losses (L BCE , L Intra , and L Inter ), and unsupervised loss (L SFL ). Particularly, weakly-supervised losses are applied to D W and D F , and unsupervised loss is applied to D U and D W . Note that there were works on the NIH dataset [27] leveraging two types of annotations [13,16,18,34] for lesion localization. However, these methods localize lesions with attention maps, which cannot be easily compared to bounding box-based detection model that we aim to develop.
β is set to 0 for the first step otherwise 0.7, δ is set to 1, λ is set to 0.99 by tuning on the validation set. As suggested in [26], α is set to 0.9, is 0.05, and γ is set to 2. CXR images are resized to 512×512 without cropping. Data augmentation is done following [33]. Adam [10] is used as the optimizer. The learning rate is initially set to 1e-5 and divided by ten when validation mAP stagnate. All implementations use Pytorch [19] on an NVIDIA TITAN Xp GPU.

Dataset and Evaluation Metrics
In total, 32,261 frontal CXR images taken from 27,253 patients were used. A cohort of 10 radiologists (4-30 years of experience) was involved in labeling the lesion bounding boxes, and each image was labeled by two with corresponding text report. If the initial labelers disagreed on the annotation, a senior radiologist (≥ 20-year experience) would make the final decision. Nine thoracic diseases including aortic calcification, cardiomegaly, fracture, mass, nodule, pleural effusion, pneumonia, pneumothorax, and tuberculosis were finally chosen in this study by the consensus of the radiologists. The dataset was split into training, validation, and testing sets with 13,963, 5,118, and 13,180 images, respectively, without patients overlapping. Detailed numbers of images and annotations per set can be found in the supplementary.
We used Average Precision (AP) [3] as the evaluation metric. Specifically, we reported the mean AP (mAP) from AP 40 to AP 75 with an interval of 5, following the Kaggle Pneumonia Detection Challenge 1 and [4]. We also evaluated AP S , AP M , and AP L for small, medium, and large targets, respectively, using the COCO API 2 . All reported statistics were averaged over the nine abnormalities.

Comparison with Other Methods
To our best knowledge, few previous works simultaneously leveraged fully-labeled data, weakly-labeled data, and unlabeled data. Hence, we first trained a Reti-naNet (with pre-trained Res101 weight from ImageNet [2]; a comparison of baselines could be found in the supplementary) on our dataset. Then, we implemented several state-of-the-art semi-supervised methods finetuned from RetinaNet, including RetinaNet [15], Π Model [11], Mean Teacher [24], MMT-PSM [33], and FocalMix [26]. We further enabled learning from the weak annotations by adding a global classification head to each semi-supervised method to construct multitask (MT) models.
Quantitative Results. All semi-supervised methods (row 2 to 5 in Table  1) clearly outperform the RetinaNet baseline (1st row), demonstrating effective utilization of the unlabeled data. After incorporating the global classification heads, the four multi-task networks (row 6 to 9) get further improvement with about 0.2∼0.4 points raising in mAP. This finding suggests that large imagelevel supervision can help learning abnormalities detection, but the benefits are still limited without proper design. On the other hand, OXnet achieves 22.3 in mAP and outperforms the multi-task models on various sizes of targets. The results corroborate that our proposed method can more effectively leverage the less well-labeled data for the thoracic disease detection task.
Ablation Study. The last four rows in Table 1   Qualitative Results. We also visualize the outputs generated by RetinaNet, Mean Teacher + MT (the best method among those compared with ours), and OXnet in Fig. 3. As illustrated, our model yields more accurate predictions for multiple lesions of different diseases in each chest X-ray sample. We also illustrates more samples of the attention maps in the supplementary.

Omni-supervision under Different Annotation Granularities
Efficient learning is crucial for medical images as annotations are extremely valuable and scarce. Thus, we investigate the effectiveness of OXnet given different annotation granularities. With results in Table 2, we find that: (1) Finer annotations always lead to better performance, and OXnet achieves consistent improvements as the annotation granularity becomes finer (row 1 to 9); (2) Increasing fully labeled data benefits OXnet more (mAP improvements are 2.9 from row 4 to 5 and 4.5 from row 5 to 6) than RetinaNet (mAP improvements are 2.1 from row 1 to 2 and 3.8 from row 2 to 3); and (3) With less fully-labeled data and more weakly-labeled data, OXnet can achieve comparable performance to RetinaNet (row 2 vs. row 4, row 3 vs. row 5). These findings clearly corroborate OXnet's effectiveness in utilizing as much available supervision as possible. Moreover, unlabeled data are easy to acquire without labeling burden, and weakly-labeled data can also be efficiently obtained with natural language processing methods [27,9,1]. Therefore, we believe OXnet could serve as a promisingly feasible and general approach to real-world clinic applications.

Conclusion
We present OXnet, a deep omni-supervised learning approach for thoracic disease detection from chest X-rays. The OXnet simultaneously utilizes well-annotated, weakly-annotated, and unlabeled data as a unified framework. Extensive experiments have been conducted and our OXnet has demonstrated superiority in effectively utilizing various granularities of annotations. In summary, the proposed OXnet has shown as a promisingly feasible and general solution to real-world applications by leveraging as much available supervision as possible.