{'fraction_non_alphanumeric': 0.033953935827061285, 'fraction_numerical': 0.010752079678569407, 'mean_word_length': 5.047912388774812, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
BSM loss: A superior way in modeling aleatory uncertainty of fine_grained classification

Artificial intelligence(AI)-assisted method had received much attention in the risk field such as disease diagnosis. Different from the classification of disease types, it is a fine-grained task to classify the medical images as benign or malignant. However, most research only focuses on improving the diagnostic accuracy and ignores the evaluation of model reliability, which limits its clinical application. For clinical practice, calibration presents major challenges in the low-data regime extremely for over-parametrized models and inherent noises. In particular, we discovered that modeling data-dependent uncertainty is more conducive to confidence calibrations. Compared with test-time augmentation(TTA), we proposed a modified Bootstrapping loss(BS loss) function with Mixup data augmentation strategy that can better calibrate predictive uncertainty and capture data distribution transformation without additional inference time. Our experiments indicated that BS loss with Mixup(BSM) model can halve the Expected Calibration Error(ECE) compared to standard data augmentation, deep ensemble and MC dropout. The correlation between uncertainty and similarity of in-domain data is up to -0.4428 under the BSM model. Additionally, the BSM model is able to perceive the semantic distance of out-of-domain data, demonstrating high potential in real-world clinical practice.


Introduction
Deep learning models have achieved enormous advances in a variety of research fields, as for medical image analysis and computer aided diagnosis [1][2][3][4]. In many situations, over-parametrized models are easily vulnerable to calibration issues, especially in the case of insufficient data and inherent noises (including label noises, instruments and equipment noises, and manual acquisition noises, etc.) [5]. Moreover, the widely used cross entropy loss with softmax activation causes disaster "over-confident" predictions [6]. It is essential to improve the accuracy of model, but the neural network "know what they don't know" has become increasingly important in clinical practice. While many types of research focused on improving the accuracy of the predictions, the uncertainty calibration (UC) of the predictions mostly remains unnoticed.
To overcome calibration deterioration, aleatoric uncertainty and epistemic uncertainty have been proposed [7][8][9] to quantify uncertainty, such that uncertain referrals can be transferred to human experts. Epistemic uncertainty is also known as model uncertainty and can be explained away given enough data. Aleatoric uncertainty captures noise inherent in the observations, corresponding to the input-dependent (data) uncertainty. Aleatoric uncertainty is irreducible, even with more training data [8,10].
Bayesian neural networks (BNNs) replace the weight parameters of the deterministic network with distributions over these parameters [11,12]. They are easy to formulate, but difficult to perform inference in [8]. In recent years, research have shown an increased interest in modeling uncertainty, such as ensemble approaches(Averaging, Voting [13], Bagging [14], Gradient Boosting [15]) , and Monte Carlo (MC) Dropout [16]. They all performed exceedingly well on large visual datasets, but sometimes badly on representation learning tasks in low-data regimes [7]. Ensemble is also typically costly with many networks trained and inferred in parallel.
For medical image processing, due to the divergence in medical equipment and acquisition process, we can hardly reduce the inherent noises by adding more training data [5,8]. Therefore, we believe that modeling the aleatoric uncertainty may be more imperative in low-data clinical practice. Test-time augmentation is a commonly used method to modeling data uncertainty during testing, and it costs much more inference time than the single forward network. Ayhan, M. S. et al. [17] explored the uncertainty estimation and the calibration of TTA in diabetic retinopathy detection and showed well calibrated measures of uncertainty in clinical practice.
Another essential factor that is needed for a proper uncertainty-aware algorithm is being distanced aware and domain-aware. It means that the algorithm distinguishes when the test data are far from those it was trained on [18]. However, the representation of distance between two datasets were not discussed in many works.
This paper studies the automatic diagnosis of the benign and malignant datasets, including the Skin Cancer data set and the clinical Breast Ultrasound data set. Based on empirical observations, we studied the trade-offs between the data-dependent uncertainty and model-dependent uncertainty. Deep Ensemble, TTA algorithms are applied to the Skin data set separately and simultaneously, and MC Dropout is also discussed because it is also one of the classic methods for modeling model uncertainty. We also compare the effects of different percentages of training data on model performance in a data-dependent way. In addition, we proposed a new method to modeling data-dependent uncertainty(both label noise and data noise) during training, using commonly data augmentation, TTA as the comparison. Besides, we consider the popular metrics for evaluating predictive uncertainty, e.g. ECE, NLL, and Brier score [7,19]. Calibration can reflect the noise modeling of the model, but the perception of semantic distance and distributed transformation are equally important. We compared the sensitivity to data distribution and domain transformation for data-dependent uncertainty. To the best of the author's knowledge, we are the first to perform uncertainty-aware on both indistribution and out-of-distribution data on small datasets of medical images.
The rest of this paper is organized as follows. Section 2 illustrates the uncertaintyaware algorithms proposed in this study. The Experimental details, including dataset introduction, parameters setting, uncertainty quantification and metrics, are described in Section 3. In Section 4, we showed the result of different methods, in addition to our Empirical Observations about deep ensemble, TTA and Mc dropout on the Skin dataset, we focused on comparing different data-dependent uncertainties calibration and perception on different dataset under different noise levels.

Mixup
Despite being robust to small amounts of label noise, the loss function of convolutional neural network is easy to overfit the noise label, especially for the relatively small amount of medical images [20]. Mixup [21] has recently demonstrated outstanding robustness against label noise without explicitly modeling. For two feature-target vectors ( , ) and ( , ) drawn at random from the training data, the loss function can be defined as drawn from a fixed mixing distribution often chosen as ( , ) Beta  where =  .

Bootstrapping loss
We first consider BS loss [22] in modeling aleatoric uncertainty. Different from focal loss, cross-entropy loss(CE loss) and other empirical risk function, BS loss maintains a higher cross-entropy loss for noise samples during training [23]. The Beta Mixture Model (BMM) better approximates the loss distribution for decision boundaries that transform linearly from clean to noisy samples. The static BS loss adds weighted labels and predicted values to the standard cross-entropy loss that helps to correct the training objective function: where i y denotes the label and i w weights the model prediction label i z , i h denotes the prediction probability on the predict class of the model.

Bootstrapping loss with Mixup (BSM)
We propose to fuse Mixup(Eq. 1.) and hard bootstrapping(Eq. 4.) to implement a robust per-sample loss correction approach and provide a smoother estimation of uncertainty: The weights i w and j w that control the proportion of labels and predictions are inferred from beta mixture model (BMM) model, Because the noise sample often shows higher cross-entropy loss than that of clean ones during training, we can infer from the loss value if a sample is more likely to be clean or noisy [23]. Supposed that the BMM model is estimated after each training epoch using the cross-entropy loss This method model the data-dependent uncertainty during training, which can save much inference time than modeling uncertainty during test time.

3
Experimental details

Datasets
We evaluated and compared methods using two collections of images:(1) Skin Cancer data set from Kaggle [24] which contains 1800 benign cases and 1497 malignant cases.
(2) Breast ultrasound of benign and malignant data set: it is collected from clinical, including 2064 benign lesions and 2064 malignant lesions acquired from Philips, Mindray, Hitachi, GE, etc. Five sophisticated doctors labeled the data independently, each of whom also examined the data labeled by others. All datasets were divided into training sets and validation sets with a 4:1 ratio. All methods used the same backbone of the modified Resnet-18 in this paper( Fig. 1. backbone).

Parameters setting
The weights of networks are initialized with Kaiming initialization [25] and we use weight decay with λ= 5e-4. We trained this module from scratch, depending on the distribution gap between natural and medical images. Empirically, we proved it better than pretrained trick and other fine-tuning operations related to natural images. We trained the model using an Adam optimizer. The initial learning rate is 5e-4 and it decays exponentially with a rate of 0.95 by epoch. Meanwhile, the minibatch size is fixed to 32, which is not a sensitive parameter in the proposed architecture. The mentioned results in this paper are performed by the most accurate model on validation data, using early stopping with "max" mode and 'patience'=20. The configuration of best performance is saved for inference. Our source code is based on PyTorch1.6 and all experiments are conducted on a server with two GeForce GTX 2080Ti GPUs. Averaging Ensemble. We averaged the outputs of multiple modified Resnet-18 for ensemble as shown in Fig. 1. All models were randomly initialized as different submodels. We followed previous works [9,26,27] and used the entropy of the mean prediction of multiple models to estimate the prediction uncertainty as below:  TTA. We adopted image augmentation model following Wang, G. et al [28]. Assuming prior distributions over the parameters of transformations, we approximated the distribution of observations 0 () px as that of augmented images () px. For medical images, one should refrain from excessive transformations that might severely alter the structure in the data [17]. We mainly use the following operations in our data augmentation pipeline during both training and testing: (1) Color jitter. Brightness: Mc Dropout. We followed Yarin Gal et al. [16]. The predictive mean and variance of the model can be written as follows: In the above equations, x is the input, ˆ() t yx is the corresponding output and  is a constant value defined based on the model structure and T is the number of stochastic forward passes during the test time.
Mixup. Compared to Ensemble and TTA, Mixup is a data augmentation technique that implements only during training and it costs the least to be inferred during the test time.
We try different  ( Table 1 in Supplements) and finally choose 0.3  = for all experiments. This may simplify data augmentation, due to the less complex structure of medical data. We also find that high values of   = may cause the deterioration in the calibration(e.g.

32
  = = ,  tends to be close to 0.5, see Table 1 in Supplements for details).

Uncertainty quantification
In this paper we used entropy for uncertainty estimation(Eq. 5) [18].
where i t is the target(for two classification task, +1 is the positive class while -1 represents the negative class) and the i y is the prediction class. So here we actually calculated the true class probability entropy(probability of prediction on label). It indicates how the NLL becomes infinite as the predictor becomes increasingly certain when test point was misclassified [19,29]. Brier score(BS). It is another metric commonly used to evaluate the model calibration and penalize both the over-confidence and the under-confidence. It is defined as: = , and 0 otherwise [19,30].

Empirical Observations
Classifier performance. Fig. 2. shows the classifiers' performance of varying amounts of ensemble models on skin cancer dataset. We applied various degrees of TTA to the ensemble models, from 0~128(see Fig. 1. in Supplements). Here we discussed TTA and deep ensemble separately and simultaneously because we think the former is a datadependent uncertainty prediction and the latter is about the model-dependent and we also want to explore the influence of the combination of the two on the model results.
We discovered that the Area Under the Curve(AUC) generally declines as we expanded the number of integrated models. It implies that randomly initialized ensemble is not always beneficial and not always makes sense. Although there are many ways to ensemble, it does not work in all the cases, especially for small datasets. In addition, we trained models with two levels of augmentation, "tta0.5" and "tta0.9", which means that we transformed data with a 50% and 90% probability of color and geometric transformation respectively. We discovered that classifiers gained moderate improvement with the increasing TTA numbers while significant improvement for the increase of training augmentation proportion. Therefore, we supposed modeling data-dependent uncertainty during training may be more important in low-data regime.
Furthermore, we observed the calibration performance and the accuracy (classification threshold is set to 0.5) of the model. It shows that randomly initialized ensemble models are generally overconfident (Fig. 2. in Supplements) and gain no superior discriminator performance (NLL and brier, see Table 1.). Additionally, we discovered that TTA had a positive effect on reliability(ECE) but no significant effect on NLL and Brier score (Fig. 3. in Supplements). The results of MC dropout are shown in Table 2. As the number of forward propagation times increasing, the AUC, NLL, and Brier score improve while the ECE decline. Decision referral. We analyzed how the number of ensembles and TTA modify the referral of decisions. Results of the study are presented in Fig. 3(a) and Fig. 3(b). It indicates a positive effect in TTA while no or negative effects in ensemble. So we further improved our hypothesis that modeling aleatoric uncertainty may be more effective in low-data regimes.

Calibrations and noise robustness
For aleatoric uncertainty, we evaluated the robustness of TTA, Mixup and BSM to inherent noise and artificial noise by manually adding 0% label noise and 20% label noise during training. The experimental results reveal that (1) TTA gains a better calibration, lower NLL and brier score than models without augmentation. (2) Table 2).

Distance perception
In order to gain additional insights into the distance perception of various networks, as well as the influence of Mixup and bootstrapping strategy, we explore the correlation between uncertainty and the distance to the (small) training set. Since there is no straightforward and semantically meaningful distance between images, we use the cosine distance between the 512-dimensional representations of the features as the similarity [7,31].  Fig. 4. First row). Here, Spearman Correlation Coefficient [32] is used to measure the rank correlation between the two variables. Table. 4. implies that the farther away a prediction is from an observation point, the higher the uncertainty gains. In addition, we considered the influence of domain transformation on the uncertainty. We used the weights of models previously trained for breast ultrasound dataset to predict the skin cancer validation sets. As is shown in the second row of Fig. 5., we discovered that Mixup and BSM model generally sustained high uncertainty for domain transfer data and was able to perceive the semantic distance of data. It is highly sensitive for preventing out-of-distribution data attacking in disease diagnosis, reflecting the "cognition" to the "unknown". Although TTA has good distance awareness for in-domain data, it is helpless for the out-of-domain.

Conclusion
It is of great significance for computer-aided diagnosis to explicitly modeling uncertainty in low-data regimes. In this paper, we introduced the BS loss combined with Mixup that modeling data-dependent uncertainty on small clinical data sets and calibrate model to lower ECE, NLL and Brier score than conventional data augmentation ,deep ensemble and MC dropout. It provided a more reliable method to learn sensible representation and successfully exhibit much robustness to label noise. As we further illustrate, Mixup is more responsive to domain-variant latent distribution and the semantic distance than the TTA method. Compared with Mixup, the BSM model consider data noise as well as label noise, which is more profitable to noise attack and outof-distribution detection. In summary, this paper verified the importance of data-dependent uncertainty modeling on medical images with small data sets and proposed a more reliable and low-risk BSM model for clinical diagnosis.

ETHICS APPROVAL AND CONSENT To PARTICIPATE
The collection and analysis of subjects' breast ultrasound data in this study were approved by the     TTA degrees (TTA0, TTA1, TTA4, TTA8, TTA16, TTA32, TTA64, TTA128) and plot the error.