{'fraction_non_alphanumeric': 0.03171289392583756, 'fraction_numerical': 0.01103489412466448, 'mean_word_length': 4.836008508992458, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Surpassing the Human Accuracy: Detecting Gallbladder Cancer from USG Images with Curriculum Learning

We explore the potential of CNN-based models for gallbladder cancer (GBC) detection from ultrasound (USG) images as no prior study is known. USG is the most common diagnostic modality for GB diseases due to its low cost and accessibility. However, USG images are challenging to analyze due to low image quality, noise, and varying viewpoints due to the handheld nature of the sensor. Our exhaustive study of state-of-the-art (SOTA) image classification techniques for the problem reveals that they often fail to learn the salient GB region due to the presence of shadows in the USG images. SOTA object detection techniques also achieve low accuracy because of spurious textures due to noise or adjacent organs. We propose GBCNet to tackle the challenges in our problem. GBCNet first extracts the regions of interest (ROIs) by detecting the GB (and not the cancer), and then uses a new multi-scale, second-order pooling architecture specializing in classifying GBC. To effectively handle spurious textures, we propose a curriculum inspired by human visual acuity, which reduces the texture biases in GBCNet. Experimental results demonstrate that GBCNet significantly outperforms SOTA CNN models, as well as the expert radiologists. Our technical innovations are generic to other USG image analysis tasks as well. Hence, as a validation, we also show the efficacy of GBCNet in detecting breast cancer from USG images. Project page with source code, trained models, and data is available at https://gbc-iitd.github.io/gbcnet


Introduction
According to GLOBOCAN 2018 [10], worldwide about 165,000 people die of GBC annually. For most patients, GBC is detected at an advanced stage, with a mean survival rate for patients with advanced GBC of six months and a 5-year survival rate of 5% [28,41]. Detecting GBC at an early stage could ameliorate the bleak survival rate.
Lately, machine learning models based on convolutional neural network (CNN) architectures have made transformational progress in radiology, and medical diagnosis for diseases such as breast cancer, lung cancer, pancreatic cancer, and melanoma [3,5,16,17,29]. However, their usage is conspicuously absent for the GBC detection. Although there has been prior work involving segmentation and detection of the GB abnormalities such as stones and polyps [14,31,35], detection of GBC is missing from the list. A search on Google Scholar with keywords "artificial intelligence" and "gallbladder cancer" returned 204 articles between 2015-2021 (November). In these, we did not find any published article on deep learning-based GBC detection from USG images. Early diagnosis and resection are critical for improving the survival rate of GBC. Due to the non-ionizing radiation, low cost, portability, and accessibility, USG is a popular diagnostic imaging modality. Although identifying anomalies such as stones or GB wall thickening at routine USG is easy, accurate characterization is challenging [25,27]. Often, USG is the sole diagnostic imaging performed for patients with suspected GB ailments. If malignancy is not suspected, no further testing is usually performed, and GBC could silently advance. Therefore, it is imperative to develop and understand the characterization of GB malignancy from USG images.
There are significant challenges in using CNN models for USG image analysis. Unlike MRI or CT, USG images suffer from low imaging quality due to noise and other sensor artifacts. The views are also not aligned due to the handheld nature of the sensor. We observe that modern CNN classifiers fail to localize the salient GB region due to the presence of shadows which often have similar visual traits of a GB in USG images (Fig. 1). Training object detectors for GBC detection gets biased towards learning from spurious textures due to noise and adjacent organ tissues rather than the shape or boundary of GB wall, which results in poor accuracy. Further, unlike normal and benign GB regions, which have regular anatomy, malignant cases are much harder to detect due to the absence of a clear GB boundary or shape and the presence of a mass.

Contributions:
The key contributions of this work are: (1) We focus on circumventing the challenges for automated detection of GBC from USG images and propose a deep neural network, GBCNet, for detecting GBC from USG images. GBCNet extracts candidate regions of interest (ROIs) from the USG to mitigate the effects of shadows and then uses a new multi-scale, secondorder pooling-based (MS-SoP) classifier on the ROIs to classify gallbladder malignancy. MS-SoP encodes rich feature representations for malignancy detection. (2) Even though GBCNet shows improvement in GB malignancy detection over multiple SOTA models, the spurious texture present in an ROI bias the classification unit towards generating false positives. To alleviate the issue, we propose a training curriculum inspired by human visual acuity [33,54]. Visual acuity refers to the sharpness of visual stimuli. The proposed curriculum mitigates texture bias and helps GBCNet focus on shape features important for accurate GBC detection from USG images.

Related Work
Deep Learning for GB Abnormalities: USG imaging is an effective modality for diagnosing GBC and related GB afflictions [59]. Lien et. al [35] use a parameter-adaptive pulse-coupled neural network for GB stone segmentation in USG images. Pang et. al [40] identify GB and gallstones using a YOLOv3 model from CT images. Jeong et. al [31] uses an InceptionV3 model to classify neoplastic polyps from cropped samples of GB USG images. GBC is a serious problem affecting a significant number of people. Despite the presence of numerous studies on using deep learning on other GB-related afflictions, there is an absence of any work that applies deep learning for GBC detection on USG images. Deep Learning for USG: CNN models have been widely applied in USG imaging tasks, such as ovarian cancer detection [60], breast cancer region, mass and boundary detection [7,13,38,58,61], measuring head circumference in fetal USG images [12,50]. Curriculum Learning: Curriculum learning has been applied to different medical imaging tasks. While Jesson et. al [32] used a patch-based curriculum for lung nodule detection, Tang et. al [53] used disease severity level to identify thoracic diseases from chest radiographs. Oksuz et. al [39] proposed image corruption-based curriculum to detect motion artifacts in cardiac MRI. Texture Bias in Neural Networks: Presence of mass and a thickened GB wall are prominent indicators of GB abnormality. However, typical CNN-based architectures are biased towards textures rather than shape [24]. This may lead GBCNet to focus on soft tissue textures such as liver rather than noticing cues based on the shape and wall of the GB. Multiple works have attempted to reduce texture bias and improve the spatial understanding of a model. Geirhos et. al [24] suggest style transfer to replace the original texture of images while Brendel et. al [11] propose a method similar to a Bag of Features model to force spatial learning. Visual Acuity in Learning Models: Vogelsang et. al [54] suggest that a period of low visual acuity (blurred vision) followed by high visual acuity induces better spatial processing and also increases the receptive field in human vision. Different from our visual acuity-inspired strategy of working with input space, Sinha et. al [47] propose applying a Gaussian kernel on the output feature map of every layer of a network. The use of blurring before pooling seems to mitigate aliasing effects due to sub-sampling in the pooling layer rather than the use of visual acuity. Azad et. al [4] have integrated a Difference of Gaussian (DoG) operation into their model. Similar to [47], they end up attenuating the high frequency in the feature maps corresponding to every layer rather than the input image, for which there is no obvious biological connection known. On the other hand, our proposed visual acuity-based curriculum works in the input space and has a solid neural basis [54].

Model Architecture
The artifacts in USG images often result in multiple spurious areas in a USG image with very similar visual traits as the GB region, leading to a disappointing performance by the baseline classifiers. GBCNet selects candidate regions of interest (ROIs) from the USG to mitigate the influence of spurious artifacts like shadows and then uses a multiscale, second-order pooling-based (MS-SoP) classifier on the ROIs. Fig. 2 presents a conceptual diagram of the architecture. At test time, the detector may predict multiple ROIs overlapping with the GB. Occasionally, the detection network may fail to localize the ROIs. In this case, we pass the entire image to the classifier. The proposed MS-SoP classifier exploits a range of spatial scales and second-order statistics to generate rich features from USG images to learn the characteristics of malignancy. We run the classifier for each candidate region during inference and aggregate the predicted labels to compute the prediction for the entire image. If any of the ROIs is classified as malignant, the image as the whole is classified as malignant. If all the regions are predicted to be normal, the image is classified as normal. In all other cases, the image is predicted to be benign. Candidate ROI Selection: We used deep object detection networks to localize the GB region in a USG image. The predicted bounding boxes serve as candidate regions of interest and mitigate the adverse effect of noise and artifacts from non-GB regions during the classification. For training the ROI selection models, we use only two classesbackground and the GB region. In this stage, we only detect the GB and do not classify them as malignant or nonmalignant. In principle, it is possible to do both in a single stage, but we observed that using a separate classifier on the focused ROIs leads to better accuracy. We note that our findings regarding the superiority of using classification on focused regions, instead of over the entire image, are consistent with other recent works [6,13,21,48,55]. Prior studies demonstrate that modern object detection architectures such as YOLO [9] or Faster-RCNN [42] can detect breast lesions in USG images [13]. On the other hand, recently proposed anchor-free approaches, such as Reppoints [57], and CentripetalNet [20] can detect unconventionally sized objects such as GB. Hence, we experimented with all the above approaches for ROI selection in our framework.

MS-SoP Classifier:
Modeling higher-order statistics has gained popularity in recent years due to its enhanced ability to capture complex features and non-linearity in deep neural networks [23,34,62]. Ning et. al [38] recently used higher-order feature fusion for classifying breast lesions. They have used RGB image patches of three fixed scales at the input layer. We take the idea further and develop a novel multi-scale, second-order pooling (MS-SoP) layer to encode rich features suitable for malignant GB detection. In contrast to [38], we exploit feature maps of multiple scales in all the intermediate layers to learn a rich representation. The proposed MS-SoP layers can be conveniently plugged into any CNN backbone. The MS-SoP classifier contains 16 MS-SoP layers as the backbone, followed by global average pooling and a fully connected classification head. We use the Categorical Cross-Entropy loss to train the classifier. Multi-Scale Block: Abdominal organs can appear in significantly different sizes in a USG image based on the insonation angle or the pressure on the transducer. Perceiving information across multiple scales is thus necessary for accurate GBC detection. Recently, Gao et. al [22] have replaced the standard convolution block in the bottleneck layer with group convolution to add a multi-scale capability to the ResNet architecture. Inspired by [22], we used a hierarchy of convolution kernels on slices of feature volumes in the intermediate layers to capture multi-scale information through a combination of different receptive fields. We split a feature map volume, X ∈ R H×W×D (H, W and D are the height, width, and the number of channels, respectively), depth-wise into 4 slices, X 1 , X 2 , X 3 , and X 4 , where X i ∈ R H×W×D/4 . Each X i will generate an output split Y i . The final output, Y, is obtained by concatenating the splits. Suppose C j are 3 × 3 convolution kernels and * denotes the convolution. We get each Y i as follows: Second-order Pooling Block: Traditional average or maxpooling use first-order statistics and thus cannot capture the higher-order statistical relation between features. Inspired by the recent success of higher-order statistics in breast lesion USG [38,61], we employ the second-order pooling (SoP) mechanism to exploit the second-order statistical dependency between the multi-scale features.
For computational efficiency, we reduce the number of channels of a feature volume, X ∈ R H×W×D , to D (D < D), using 1×1 convolutions. X is then reshaped to a matrix X ∈ R D ×N , where N = H × W . We compute the covariance of X as, C D ×D = (X−X)(X−X) T , which is then reshaped to a tensor of size 1×D ×D and passed through a convolution layer with 4D kernels of size 1×D each. We resize the resulting 1×1×4D tensor to a 1×1×D tensor, Figure 3. Grad-CAM visual of GBCNet trained without curriculum showing how the model tends to get biased due to the presence of textures due to noise or organ tissue. GBCNet focuses on -(a) adjacent liver tissues than the normal GB, (b) the echogenic region below the GB, and (c) liver textures instead of the stones (highlighted using the arrow).
w d , by 1 × 1 convolutions. w d represents the weight for each channel. These weights are then channel-wise multiplied with X , to obtain the weighted feature map Z d . To repeat similar processes for the height and width, we transpose X from to a D × W × H tensor, say X h and to a H × D × W tensor, say X w . In terms of index notation, We also calculate Z h and Z w by multiplying w h and w w channel-wise to X h and X w . Then, we transpose Z h and Z w to tensors of . Finally, we obtain the output feature tensor, Z ∈ R H×W×D by adding Z d , Z h T , and Z w T .

Visual Acuity Inspired Curriculum
We found that the textures having visual characteristics of soft tissue can adversely affect the performance of GBC-Net (Fig. 3). We propose a curriculum to mitigate the texture bias and improve the classification. We observed that while the MS-SoP classifier is affected by texture bias, the region selection network still maintains a very high recall (Tab. 2). Hence, we used the curriculum training only on the classifier and not on the region selection network. Visual Acuity in Humans: Visual Acuity (VA) refers to the clarity and sharpness of human vision. Due to the immaturity of the retina and visual cortex, newborn children have very low VA [18]. The VA improves with the maturation of the retina and visual cortex. However, for children with congenital cataracts, the cortex matures despite the lenticular opacity. Such children begin their visual activity with higher initial VA. Evidence shows that children with high initial VA suffer to facilitate spatial analysis over expansive areas [54]. Low VA renders blurry images that do not contain enough local information for the visual cortex to identify patterns. As a result, the visual cortex tries to increase the receptive field to facilitate spatial analysis over expansive areas and learn global features [33,49]. Gaussian Blurring to Simulate Visual Acuity: Gaussian filters are low-pass filters to cloak the high-frequency components of an input. A standard deviation σ parameterizes the Gaussian filters. Increasing the σ generates a higher amount of blur and low VA when convolved with an image. Fig. 4 shows how we can decrease the VA by increasing the σ of a Gaussian filter. In our experiments, we have varied σ from 1 to 16 to generate different levels of VA. Proposed Curriculum: While [54] demonstrates the improvement in receptive fields by gradually improving the sharpness of images during training, we take this observation further and show that the strategy of training on progressively higher resolution images also reduces the texture bias of a classification model. We propose a visual acuitybased training curriculum (Algorithm 1) that starts training the network with blurry and low-resolution USG images and progressively increase the sharpness of training samples. The initial blurring allows the model to use an extended receptive field and focus on learning the global features such as the shape of the GB while ignoring any noise or irrelevant textures. In the later phases, the sharp images allow the model to focus on the relevant local features in a controlled manner to make more accurate predictions.

Dataset Collection and Curation
Data Collection: We acquired data samples from patients referred to PGIMER, Chandigarh (a tertiary care referral hospital in Northern India) for abdominal ultrasound examinations of suspected GB pathologies. The study was approved by the Ethics Committee of PGIMER. We obtained informed written consent from the patients at the time of recruitment, and protect their privacy by fully anonymizing the data. Minimum 10 grayscale B-mode static images, including both sagittal and axial sections, were recorded by radiologists for each patient using a Logiq S8 machine. We excluded color Doppler, spectral Doppler, annotations, and measurements. Supplementary A contains more details of the data acquisition process. Labeling and ROI Annotation: Each image is labeled as one of the three classes -normal, benign, or malignant. The ground-truth labels were biopsy-proven to assert the correctness. Additionally, in each image, expert radiologists have drawn an axis-aligned bounding box spanning the entire GB and adjacent liver parenchyma to annotate the ROI. Dataset Statistics: We have annotated 1255 abdominal USG images collected from 218 patients from the acquired image corpus. Overall, we have 432 normal, 558 benign, and 265 malignant images. Of the 218 patients, 71, 100, and 47 were from the normal, benign, and malignant classes, respectively. The width of the images was between 801 and 1556 pixels, and the height was between 564 and 947 pixels due to the cropping of patient-related information. Dataset Splits: The sizes of the training and testing sets are 1133 and 122, respectively. To ensure generalization to unseen patients, all images of any particular patient were either in the train or the test split. The number of normal, benign, and malignant samples in the train and test set is 401, 509, 223, and 31, 49, and 42, respectively. Additionally, we report the 10-fold cross-validation metrics on the entire dataset for key experiments to assess generalization. All images of any particular patient appeared either in the training or the validation split during the cross-validation. . We also report the GBC detection performance of two expert radiologists on the test set. The radiologists classified each test image without accessing the biopsy results or any other patient data. Note that our model significantly outperforms even the human radiologists. Recall that our ground truth labels are biopsyproven. The performance of human radiologists in the our study is comparable to that reported in literature [8,26].

Implementation and Evaluation
Transfer Learning and Data Augmentation: Studies show that pre-training on natural image data improves network performance on medical image data [2,15]. We use ROI detection networks pre-trained on the COCO [37], and classifiers pre-trained on ImageNet data [19]. We use resizing, center-cropping, and normalization data augmentations to avoid over-fitting on the small USG dataset. Hyper-parameters: Details of the hyper-parameters are in the supplementary C. We train and freeze the ROI detection network before training the classifier. We used σ 0 = 16, k = 10, k = 5, for the curriculum. Evaluation metrics: We use mean intersection over union (mIoU), precision, and recall for comparing region selection models. We compute the precision and recall as suggested by [43] (details in the supplementary D). To assess the classification models for GBC detection, we use accuracy, sensitivity, and specificity as the evaluation metrics.

Efficacy of GBCNet over Baselines
We compare GBCNet with three popular deep classifiers, ResNet-50 [30], VGG-16 [46], and Inception-V3 [51]. We also evaluate the performance of three SOTA object detectors, Faster-RCNN [42], RetinaNet [36], and Efficient-Det [52] for detecting GBC. We report the results in Tab. 1. From the reported results, it is clear that baseline networks have poor accuracy for detecting GBC from USG images. Grad-CAM [45] visualizations in Fig. 5 show that the noise,  textures, and artifacts significantly influence the decision of baseline classification models. As a result, their classification accuracy suffers heavily. Compared to the baselines, GBCNet along-with the proposed MS-SoP classifier precisely focuses on crucial visual cues leading to its superior performance.

Performance of GB Region Selection Models
Tab. 2 summarizes the performance of various models for localizing the GB region. For critical tasks such as region selection for cancer detection, recall is more important Figure 6. We visually compare ROI selection by Faster-RCNN (dark red) with the ROI identified by expert Radiologists (light yellow). (a, b) The predicted aROI matches well with the radiologists' expectations. (c) The model considers the sections partitioned by the GB wall as separate regions. However, the union of the predicted boxes very closely approximates the actual GB region. (d, e) Although the radiologist made an error in judging the ROI, Faster-RCNN was able to identify an accurate ROI resulting in a visually superior prediction. (f) The predicted ROI covers only a portion of the area an expert radiologist considered necessary. Even though the region prediction seems inferior compared to the human perception, expert radiologists corroborated that the predicted region captures the GB invading the liver, a vital visual cue to detect GBC. ROI samples from other detectors are in supplementary F.

Model
Spec. Sens.  Table 3. The sensitivity and specificity of MS-SoP and two baseline classifiers on breast cancer detection from USG images. We report 5-fold cross-validation on the BUSI dataset. than precision. Multiple predicted regions can be discarded in the second stage, but missing any potentially malignant region could be disastrous. We note that the Faster-RCNN achieves the highest mIoU out of all the models while maintaining very high recall and excellent precision. Hence, we use Faster-RCNN as the region selection model. In Fig. 6 we show the visual comparison of the GB localization results of Faster-RCNN along with the ROIs annotated by the expert radiologists. The model could predict the region of interest accurately in most cases. Although the model's prediction visually differed from the radiologists in some samples, closer inspection revealed that the predicted region retains sufficient visual cues to detect malignancy.

Applicability of the Proposed Classifier in Breast Cancer Detection from USG Images
We explored the applicability of the proposed MS-SoP classifier on breast cancer detection from USG images for  Table 4. Robustness of the curriculum in tacking texture bias while detecting GBC. We show the performance of using curriculum on four models that apply classifiers on localized GB region -(a) ROI+VGG16, (b) ROI+ResNet50, (c) ROI+Inception-V3, and (d) GBCNet (ROI+MS-SoP). The relative change (in percentage) in specificity for synthetic test data is shown within parentheses. The sensitivity remains unchanged as the malignant images were not altered. Observe that as compared to the models trained on highresolution images, our VA-based curriculum is more robust to textures and is able to maintain a lower drop in specificity. The only exception is the ROI+VGG16 model, for which the curriculum training does not lower the drop in specificity.
a publicly available dataset, BUSI [1], containing 133 normal, 487 benign, and 210 malignant images. The images in BUSI are already cropped from original USG images to highlight only the important regions. Thus, we skip the ROI selection and run the MS-SoP classifier on BUSI. Tab. 3 shows that the MS-SoP classifier achieves much better sensitivity, which indicates the superiority of the MS-SoP architecture for malignancy identification on USG images. We note that while breast cancer detection relies on tissue/ mass characterization, GBC detection is primarily based on wall shape and mass anomaly.

Efficacy of the Proposed Curriculum
Robustness in Tackling Texture Bias: As described earlier, spurious textures present in USG images tend to increase the false positives in detecting malignancy. To validate this hypothesis, we created a synthetic test set. We used the method given by [56] and added low-level frequencies from malignant images to alter the original texture of normal and benign samples. We also manually added patches looking like soft tissue near the GB region of normal and benign images. Expert radiologists confirmed that the diagnosis of the GB pathology is not altered. Tab. 4 shows that the specificity decreases due to the increase of the false positives. The sensitivity remains unaffected as the prediction for malignant GB samples was unchanged. The network trained with the proposed curriculum tackles texture bias well and accurately predicts non-malignant samples in synthetic test data. This experiment shows that the proposed curriculum effectively tackles texture bias. Fig. 7 shows a visual sample of how the soft tissue-like texture influences the network's decision and how the curriculum helps the network to rectify the discriminative regions.
Performance Improvement: We also assess the quantitative performance improvement of models due to the curriculum training in supplementary B.

Ablation Study
Choice of Classifier in GBCNet: We have plugged in other deep classification networks in place of the proposed MS-SoP classifier in the GBCNet framework. Fig. 8a summarizes the results. The MS-SoP classifier on GBCNet provides the best GBC detection accuracy. Using the classifiers on the ROIs improves the sensitivity and accuracy for ResNet50 and VGG16. However, the drop in specificity results in performance degradation for Inception-V3 as the sensitivity was not improved. Choice of Training Regime: We used the proposed GBC-Net model to assess the influence of the visual acuity-based curriculum. We compare the curriculum with two possible alternatives -(i) anti-curriculum that initially trains with high resolution and progressively lowers the resolution, and (ii) control-curriculum where the samples are not sorted resolution-wise, and the curriculum contains a random set of blurred samples. Note that the control-curriculum can also be thought of using Gaussian blurring as a data augmentation where the probability of choosing a particular σ is equal to the fraction of epochs the σ used during the curriculum. Fig. 8b and 8c show the performance of the various curriculum strategies on the performance on GBCNet. Fur- ther, to understand how various curriculum strategies affect a model's generalization at different resolutions, we blur the test set using different values of σ and evaluate the models on these images (Fig. 8d). We see that the model trained using the proposed curriculum generalizes well across different image resolutions, which is an indicator of better spatial understanding.

Conclusion
This paper addresses Gallbladder Cancer detection from Ultrasound images using deep learning and proposes a new supervised learning framework (GBCNet) based on ROI selection and multi-scale second-order pooling. The proposed design helps the classifier focus on the crucial GB region predicted by the region selection network. We propose a visual acuity-based curriculum to make our design resilient to texture bias and improve its specificity. Extensive experiments show that GBCNet, combined with curriculum learning, improves performance over the baseline deep classification and object detection architectures. We hope our work will generate interest in the community towards this important but hitherto overlooked problem of GBC detection. Table S1. Model performances (10-fold cross-validation) for training with our proposed visual acuity-based curriculum.

C. Implementation details
Tab. S2 lists the configurations of all models which we have used. We trained on the Quadro P5000 16GB GPU. The table includes a brief description of the various stages of the network, input image sizes (H × W × D), the optimizer, relevant hyper-parameters such as learning rate, weight decay, momentum, batch size, and the number of training epochs/steps for the network.

D. Calculating Precision and Recall for GB Localization Networks
For computing precision and recall during the GB localization phase, as suggested by [43], if the center of the predicted region lies within the bounding box of the ground truth region, then we consider a region prediction to be a true positive; otherwise, we consider the region prediction to be a false positive due to localization error. Further, we consider the zero/no region prediction as a false negative (all our images contain GB, and the localization network's task is to merely localize it). Figure S2 shows the sample Grad-CAM visualizations of the predictions using GBCNet (ROI+MS-SoP) with curriculum learning.

F. ROI Visuals
In figure S3, we show sample predictions of the GB region localization for different models. We also show the region of interest as perceived by the expert radiologists. The localization model is fairly accurate in capturing important regions of the USG image.