{'fraction_non_alphanumeric': 0.04005162905307623, 'fraction_numerical': 0.017913716900692298, 'mean_word_length': 4.466060929983966, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Calibrating Deep Neural Networks using Focal Loss

Miscalibration -- a mismatch between a model's confidence and its correctness -- of Deep Neural Networks (DNNs) makes their predictions hard to rely on. Ideally, we want networks to be accurate, calibrated and confident. We show that, as opposed to the standard cross-entropy loss, focal loss (Lin et al., 2017) allows us to learn models that are already very well calibrated. When combined with temperature scaling, whilst preserving accuracy, it yields state-of-the-art calibrated models. We provide a thorough analysis of the factors causing miscalibration, and use the insights we glean from this to justify the empirically excellent performance of focal loss. To facilitate the use of focal loss in practice, we also provide a principled approach to automatically select the hyperparameter involved in the loss function. We perform extensive experiments on a variety of computer vision and NLP datasets, and with a wide variety of network architectures, and show that our approach achieves state-of-the-art accuracy and calibration in almost all cases.


Introduction
Deep neural networks have dominated computer vision and machine learning in recent years, and this has led to their widespread deployment in real-world systems [Cao et al., 2018, Chen et al., 2018, Kamilaris and Prenafeta-Boldú, 2018, Ker et al., 2018. However, many current multi-class classification networks in particular are poorly calibrated, in the sense that the probability values that they associate with the class labels they predict overestimate the likelihoods of those class labels being correct in the real world. This is a major problem, since if networks are routinely overconfident, then downstream components cannot trust their predictions. The underlying cause is hypothesised to be that these networks' high capacity leaves them vulnerable to overfitting on the negative log-likelihood (NLL) loss they conventionally use during training [Guo et al., 2017].
Given the importance of this problem, numerous suggestions for how to address it have been proposed. Much work has been inspired by approaches that were not originally formulated in a deep learning context, such as Platt scaling [Platt, 1999], histogram binning [Zadrozny and Elkan, 2001], isotonic regression [Zadrozny and Elkan, 2002], and Bayesian binning and averaging [Naeini et al., 2015, Naeini andCooper, 2016]. As deep learning has become more dominant, however, various works have begun to directly target the calibration of deep networks. For example, Guo et al. [Guo et al., 2017] have popularised a modern variant of Platt scaling known as temperature scaling, which works by dividing a network's logits by a scalar T > 0 (learnt on a validation subset) prior to performing softmax. Temperature scaling has the desirable property that it can improve the calibration of a network without in any way affecting its accuracy. However, whilst its simplicity and effectiveness have made it a popular network calibration method, it does have downsides. For example, whilst it scales the logits to reduce the network's confidence in incorrect predictions, this also slightly reduces the network's confidence in predictions that were correct.
By contrast, [Kumar et al., 2018] initially eschew temperature scaling in favour of minimising a differentiable proxy for calibration error at training time, called Maximum Mean Calibration Error (MMCE), although they do later also use temperature scaling as a post-processing step to obtain better results than cross-entropy followed by temperature scaling [Guo et al., 2017]. Separately, [Müller et al., 2019] propose training models on cross-entropy loss with label smoothing instead of one-hot labels and show that label smoothing has a very favourable effect on model calibration.
In this paper, we propose a technique for improving network calibration that works by replacing the crossentropy loss conventionally used when training classification networks with the focal loss proposed by [Lin et al., 2017]. We observe that unlike cross-entropy, which minimises the KL divergence between the predicted (softmax) distribution and the target distribution (one-hot encoding in classification tasks) over classes, focal loss minimises a regularised KL divergence between these two distributions, which ensures minimisation of the KL divergence whilst increasing the entropy of the predicted distribution, thereby preventing the model from becoming overconfident. Since focal loss, as shown in §4, is dependent on a hyperparameter, γ, that needs to be cross-validated, we also provide a method of choosing γ automatically for each sample, and show that it outperforms all the baseline models.
The intuition behind using focal loss is to direct the network's attention during training towards samples for which it is currently predicting a low probability for the correct class, since trying to reduce the NLL on samples for which it is already predicting a high probability for the correct class is liable to lead to NLL overfitting, and thereby miscalibration [Guo et al., 2017]. More formally, we show in §4 that focal loss can be seen as implicitly regularising the weights of the network during training by causing the gradient norms for confident samples to be lower than they would have been with cross-entropy, which we would expect to reduce overfitting and improve the network's calibration.
Overall, we make the following contributions: 1. In §3, we study the link that [Guo et al., 2017] observed between miscalibration and NLL overfitting in detail, and show that the overfitting is associated with the predicted distributions for misclassified test samples becoming peakier as the optimiser tries to increase the magnitude of the network's weights to reduce the training NLL. 2. In §4, we propose the use of focal loss for training better-calibrated networks, and provide both theoretical and empirical justifications for this approach. In addition, we provide a principled method of automatically choosing γ for each sample during training. 3. In §5.1, we show, via experiments on a variety of classification datasets and network architectures, that DNNs trained with focal loss are more calibrated than those trained with cross-entropy loss (both with and without label smoothing), MMCE or Brier loss [Brier, 1950]. 4. Finally, in §5.2, we show that a network trained using focal loss is able to improve on both the BLEU and ECE scores of baseline models trained using cross-entropy (with one-hot labels and label smoothing) on the WMT 2014 English-to-German translation dataset, thereby showing the practical impact of the calibration improvements we are able to achieve for performance on downstream tasks.

Problem Formulation
Let D = (x i , y i ) N i=1 denote a dataset consisting of N samples from a joint distribution D(X , Y), where for each sample i, x i ∈ X is the input and y i ∈ Y = {1, 2, ..., K} is the ground-truth class label. Letp i,y = f θ (y|x i ) be the probability that a neural network f with model parameters θ predicts for a class y on a given input x i . The class that f predicts for x i is computed asŷ i = argmax y∈Ypi,y , and the predicted confidence aŝ p i = max y∈Ypi,y . The network is said to be perfectly calibrated when, for each sample (x, y) ∈ D, the confidencep is equal to the model accuracy P(ŷ = y|p), i.e. the probability that the predicted class is correct.
where 1 is the indicator function, andŷ j and y j are the predicted and ground-truth labels for the j th sample. Similarly, the confidence C i of the i th bin is computed as C i = 1 |Bi| j∈Bip j , i.e. C i is the average confidence of all samples in the bin. The ECE can be approximated as a weighted average of the absolute difference between the accuracy and confidence of each bin: A similar metric, the maximum calibration error (MCE) [Naeini et al., 2015], is defined as the maximum absolute difference between the confidence and accuracy of each bin: A common way of visually exploring the calibration of a model is to use a reliability plot [Niculescu-Mizil and Caruana, 2005], which plots the accuracies of the various confidence bins as a bar chart (e.g. see Figure 1). Reliability plots also capture whether or not a model is under-confident or over-confident in general. For a perfectly calibrated model, the accuracy for each bin will match the confidence, and hence all of the bars will lie on a diagonal. By contrast, if most of the bars lie above the diagonal, meaning that the model is more accurate than it expects, then it is said to be under-confident, and if most of the bars lie below the diagonal, then it is said to be over-confident.
AdaECE: One disadvantage of ECE is the uniform bin width. Once a model is trained, most of the samples lie within the highest confidence bins, and hence these bins dominate the value of the ECE, as the contribution of each bin is weighted by the number of samples it contains. To mitigate this, we thus also consider another metric, which we call AdaECE (Adaptive ECE), for which bin sizes are calculated so as to evenly distribute samples between bins:

What Causes Miscalibration?
We now discuss why high-capacity neural networks, despite achieving low classification errors on well-known datasets, tend to be miscalibrated. A key empirical observation made by [Guo et al., 2017] was that poor calibration of such networks appears to be linked to overfitting on the negative log-likelihood (NLL) during training. In this section, we further inspect this observation to provide new insights.
For the analysis, we train a high-capacity ResNet-50 network on CIFAR-10 with state-of-the-art performance settings [PyTorch-CIFAR]. We use Stochastic Gradient Descent (SGD), with a mini-batch of size 128, momentum of 0.9, and learning rate schedule of {0.1, 0.01, 0001} for the first 150, next 100, and last 100 epochs, respectively. We minimise cross-entropy loss (a.k.a. NLL) L c , which, in a standard classification context, is − logp i,yi , wherê p i,yi is the probability assigned by the network to the correct class y i for the i th sample. Note that the NLL is minimised when for each training sample i,p i,yi = 1, whereas the classification error is minimised when p i,yi >p i,y for all y = y i . This indicates that even when the classification error is 0, the NLL can be positive, and the optimisation algorithm can still try to reduce it to 0 by further increasing the value ofp i,yi for each sample. This can be empirically observed in Figure 1, where we plot reliability diagrams and percentages of samples in each confidence bin for different training epochs.
To study how miscalibration occurs during training, we plot the average NLL for the train and test sets at each training epoch in Figures 2(a) and 2(b). We also plot the average NLL and the entropy of the softmax distribution produced by the network for the correctly and incorrectly classified samples. In Figure 2(c), we plot the classification errors on the train and test sets, along with the test set ECE.
Curse of misclassified samples: Figures 2(a) and 2(b) show that although the average train NLL (for both correctly and incorrectly classified training samples) broadly decreases throughout training, after the 150 th epoch (where the learning rate drops by a factor of 10), there is a marked rise in the average test NLL, indicating that the network starts to overfit on average NLL. Interestingly, the increase in average test NLL is caused only by the incorrectly classified samples, as the average NLL for the correctly classified samples continues to decrease even after the 150 th epoch. We also observe that after epoch 150, the test set ECE rises, indicating that the network is becoming miscalibrated. This corroborates the observation in [Guo et al., 2017] that miscalibration and NLL overfitting are linked.
Peak at the wrong place: We further observe that the entropies of the softmax distributions for both the correctly and incorrectly classified test samples decrease throughout training (in other words, the distributions get peakier). This observation, coupled with the one we made above, indicates that for the wrongly classified test samples, the network gradually becomes more and more confident about its incorrect predictions.
Weight magnification: The increase in confidence of the network's predictions can happen if the network increases the norm of its weights W to increase the magnitudes of the logits. In fact, cross-entropy loss is minimised when for each training sample i,p i,yi = 1, which is possible only when ||W || → ∞. Cross-entropy loss thus inherently induces this tendency of weight magnification in neural network optimisation.  Figure 3: How metrics related to model calibration change whilst training several ResNet-50 networks on CIFAR-10, using either cross-entropy loss, or focal loss with γ set to 1, 2 or 3.
performance of weight decay (regulating the norm of weights) on the calibration of neural networks can perhaps be explained using this. We explore this further in §4. This increase in the network's confidence during training is one of the key causes of miscalibration.

Improving Calibration using Focal Loss
As discussed in §3, overfitting on NLL, which is observed as the network grows more confident on all of its predictions irrespective of their correctness, is strongly related to poor calibration. One cause of this is that the cross-entropy objective minimises the difference between the softmax distribution and the ground-truth one-hot encoding over an entire mini-batch, irrespective of how well a network classifies individual samples in the mini-batch. In this work, we study an alternative loss function, popularly known as focal loss [Lin et al., 2017], that tackles this by weighting loss components generated from individual samples in a mini-batch by how well the model classifies them. For classification tasks where the target distribution is a one-hot encoding, it is defined as Why might focal loss improve calibration? We know that cross-entropy forms an upper bound on the KLdivergence between the target distribution q and the predicted distributionp, i.e. L c ≥ KL(q||p), so minimising cross-entropy results in minimising KL(q||p). Interestingly, a general form of focal loss can be shown to be an upper bound on the regularised KL-divergence, where the regulariser is the negative entropy of the predicted distributionp, and the regularisation parameter is γ, the hyperparameter of focal loss (a proof of this can be found in Appendix A): The most interesting property of this upper bound is that it shows that replacing cross-entropy with focal loss has the effect of adding a maximum-entropy regulariser [Pereyra et al., 2017] to the implicit minimisation that was previously being performed. In other words, trying to minimise it means trying to minimise the KL divergence betweenp and q, whilst simultaneously trying to increase the entropy of the predicted distribution p. Encouraging the predicted distribution to have higher entropy can help avoid the overconfident predictions produced by modern neural networks (see the 'Peak at the wrong place' paragraph of Section 3), and thereby improve calibration.
Empirical observations: To analyse the behaviour of neural networks trained on focal loss, we use the same framework as mentioned above, and train four ResNet-50 networks on CIFAR-10, one using cross-entropy loss, and three using focal loss with γ = 1, 2 and 3. Figure 3(a) shows that the test NLL for the cross-entropy model significantly increases towards the end of training (before saturating), whereas the NLLs for the focal loss models remain low. To better understand this, we analyse the behaviour of these models for correctly and incorrectly classified samples. Figure 3(b) shows that even though the NLLs for the correctly classified samples broadly-speaking decrease over the course of training for all the models, the NLLs for the focal loss models remain consistently higher than that for the cross-entropy model throughout training, implying that the focal loss models are relatively less confident than the cross-entropy model for samples that they predict correctly. This is important, as we have already discussed that it is overconfidence that normally makes deep neural networks miscalibrated. Figure 3(c) shows that in contrast to the cross-entropy model, for which the NLL for misclassified test samples increases significantly after epoch 150, the rise in this value for the focal loss models is much less severe. Additionally, in Figure 3(d), we notice that the entropy of the softmax distribution for misclassified test samples is consistently (if marginally) higher for focal loss than for cross-entropy (consistent with Equation 4).
As per §3, an increase in the test NLL and a decrease in the test entropy for misclassified samples, taken together with no corresponding increase in the test NLL for the correctly classified samples, can be interpreted as the network starting to predict softmax distributions for the misclassified samples that are ever more peaky in the wrong place. Notably, our results in Figures 3(b), 3(c) and 3(d) clearly show that this effect is significantly reduced when training with focal loss rather than cross-entropy, leading to a network that is less peaky in the wrong place and better calibrated.
Theoretical justification: As mentioned previously, once a model trained using cross-entropy reaches high accuracy on the training set, the optimiser may try to further reduce the training NLL by increasing the confidences for the correctly classified samples. One way it could achieve this would be to increase the weights of the network to increase the magnitudes of the logits. In fact, this hypothesis would help to explain the observation of [Guo et al., 2017] that models trained using some form of weight decay are relatively better calibrated. To verify this, we plot the L 2 norm of the weights of the last linear layer for all four networks as a function of the training epoch (see Figure 3(e)). Notably, although the norms of the weights for the models trained on focal loss are initially higher than that for the cross-entropy model, a complete reversal in the ordering of the weight norms occurs between epochs 150 and 250. In other words, as the networks start to become miscalibrated, the weight norm for the cross-entropy model also starts to become greater than those for the focal loss models. In practice, this is because focal loss, by design, starts to act as a regulariser on the network's weights once the model has gained a certain amount of confidence in its predictions. This behaviour of focal loss can be observed even on a much simpler setup like a linear model (see Appendix B). To better understand this, we start by considering the following proposition: Proposition 1. For focal loss L f and cross-entropy L c , the gradients is the focal loss hyperparameter, and w denotes the parameters of the last linear layer. Thus Proposition 1 shows the relationship between the norms of the gradients of the last linear layer for focal loss and cross-entropy loss, for the same network architecture. Note that this relation depends on a function g(p, γ), which we plot in Figure 4(a) to understand its behaviour. It is clear that for every γ, there exists a (different) threshold p 0 such that for all p ∈ [0, p 0 ], g(p, γ) ≥ 1, and for all p ∈ (p 0 , 1], g(p, γ) < 1. (For example, for γ = 1, p 0 ≈ 0.4.) We use this insight to further explain why focal loss provides implicit weight regularisation. Implicit weight regularisation: For a network trained using focal loss with a fixed γ, during the initial stages of the training, whenp i,yi ∈ (0, p 0 ), g(p i,yi , γ) > 1. This implies that the confidences of the focal loss model's predictions will initially increase faster than they would for cross-entropy. However, as soon asp i,yi crosses the threshold p 0 , g(p i,yi , γ) falls below 1 and reduces the size of the gradient updates made to the network weights, thereby having a regularising effect on the weights. This is why, in Figure 3(e), we find that the weight norms of the models trained with focal loss are initially higher than that for the model trained using cross-entropy. However, as training progresses, we find that the ordering of the weight norms reverses, as focal loss starts regularising the network weights. Moreover, we can draw similar insights from Figures 4(b), 4(c) and 4(d), in which we plot histograms of the gradient norms of the last linear layer (over all samples in the training set) at epochs 10, 100 and 200, respectively. At epoch 10, the gradient norms for cross-entropy and focal loss are similar, but as training progresses, those for cross-entropy decrease less rapidly than those for focal loss, indicating that the gradient norms for focal loss are consistently lower than those for cross-entropy throughout training.
Finally, observe in Figure 4(a) that for higher γ values, the fall in g(p, γ) is steeper. We would thus expect a greater weight regularisation effect for models that use higher values of γ. This explains why, of the three models that we trained using focal loss, the one with γ = 3 outperforms (in terms of calibration) the one with γ = 2, which in turn outperforms the model with γ = 1. Based on this observation, one might think that, in general, a higher value of gamma would lead to a more calibrated model. However, this is not the case, as we notice from Figure 4(a) that for γ ≥ 7, g(p, γ) reduces to nearly 0 for a relatively low value of p (around 0.5). As a result, using values of γ that are too high will cause the gradients to die (i.e. reduce to nearly 0) early, at a point at which the network's predictions remain ambiguous, thereby causing the training process to fail.
How to choose γ: As discussed, focal loss provides implicit entropy and weight regularisation, both of which heavily depend on the value of γ. Finding an appropriate γ is normally done using cross-validation. Also, traditionally, γ is fixed for all samples in the dataset. However, as shown, the regularisation effect for a sample i depends onp i,yi , i.e. the predicted probability for the ground truth label for the sample. It thus makes sense to choose γ for each sample based on the value ofp i,yi . To this end, we provide Proposition 2, which we use to find a solution to this problem: log a , where a = 1 − p 0 , b = p 0 log p 0 , and W −1 is the Lambert-W function [Corless et al., 1996]. Moreover, for p ≥ p 0 > 0 and γ ≥ γ * , the equality g(p, γ) = 1 holds only for p = p 0 and γ = γ * .
Proof. See Appendix C. Note that there exist multiple values of γ for which g(p, γ) ≤ 1 for all p ≥ p 0 .
For a given p 0 , this allows us to compute γ s.t. (i) g(p 0 , γ) = 1; (ii) g(p, γ) ≥ 1 for p ∈ [0, p 0 ); and (iii) g(p, γ) < 1 for p ∈ (p 0 , 1]. This allows us to control the magnitude of the gradients for a particular sample i based on the current value ofp i,yi , and gives us a way of obtaining an informed value of γ for each sample. For instance, a reasonable policy might be to choose γ s.t. g(p i,yi , γ) > 1 ifp i,yi is small (say less than 0.25), and g(p i,yi , γ) < 1 otherwise. Such a policy will have the effect of making the weight updates larger for samples having a low predicted probability for the correct class and smaller for samples with a relatively higher predicted probability for the correct class.
Following the aforementioned arguments, we choose a threshold p 0 of 0.25, and use Proposition 2 to obtain a γ policy such that g(p, γ) is observably greater than 1 for p ∈ [0, 0.25) and g(p, γ) < 1 for p ∈ (0.25, 1]. In particular, we use the following schedule: ifp i,yi ∈ [0, 0.2), then γ = 5, otherwise γ = 3 (note that g(0.2, 5) ≈ 1 and g(0.25, 3) ≈ 1, refer Figure 4(a)). We find it to perform consistently well across multiple classification datasets and network architectures. Having said that, one can calculate multiple such schedules for γ following Proposition 2, using the intuition of having a relatively high γ for low values ofp i,yi and a relatively low γ for high values ofp i,yi .

Image and Document Classification
We use multiple image and document classification datasets to verify the effectiveness of focal loss for training calibrated models. For our image classification experiments, we use CIFAR-10/100 [Krizhevsky, 2009] and Tiny-ImageNet [Deng et al., 2009], and for document classification, we use 20 Newsgroups [Lang, 1995] and the Stanford Sentiment Treebank (SST) [Socher et al., 2013]. We provide details regarding these datasets and their train/validation/test splits in Appendix D. On CIFAR-10/100, we train ResNet-50, ResNet-110 [He et al., 2016], Wide-ResNet-26-10 [Zagoruyko and Komodakis, 2016] and DenseNet-121 [Huang et al., 2017]. On Tiny-ImageNet, we train a ResNet-50 network. We train a Global Pooling Convolutional Network [Lin et al., 2014] on 20 Newsgroups [Lang, 1995] and a Tree-LSTM [Tai et al., 2015] on the SST Binary dataset. We provide implementation details for training each of these models in Appendix D. We use the following loss functions for training the above models: Baselines Along with cross-entropy loss, we compare our method against the following strong baselines (with and without temperature scaling): 1. MMCE (Maximum Mean Calibration Error) [Kumar et al., 2018]: a continuous and differentiable proxy for calibration error that is normally used as a regulariser alongside cross-entropy.
2. Label smoothing [Müller et al., 2019] (LS): given a one-hot ground-truth label distribution q and a smoothing factor α (hyperparameter), the smoothed vector s is obtained as , where s i and q i denote the i th elements of vectors s and q respectively, and K is the number of classes. Instead of q, the smoothed vector is now treated as the ground truth. In our experiments, we train models using smoothing factors α = 0.05 and α = 0.1 but find α = 0.05 to perform better. We thus report the results obtained from LS-0.05 with α = 0.05.
3. Brier Score [Brier, 1950]: computed as the squared error between the predicted softmax vector and the one-hot ground truth encoding. Brier score is particularly relevant baseline for calibration as it can be decomposed into calibration and refinement [DeGroot andFienberg, 1983, Snoek et al., 2019]. Moreover, it has a distinct penalty on incorrect class probabilities.
In addition to the aforementioned sample-dependent γ approach, we also train other baselines on focal loss as well. We train models on focal loss with γ fixed to 1, 2 and 3. As a simplification to the sample-dependent γ approach, we also try using a training epoch-dependent schedule for γ. We describe these in more detail and report the results in Appendix E.
Temperature Scaling: We compute the optimal temperature using two different approaches: (a) learning the optimal temperature by optimising NLL over a validation set, and (b) performing grid search over temperature values between 0 and 10, with a step of 0.1, and choosing the temperature that minimises the validation set ECE. We find the second approach to produce stronger baselines. Since we report ECE and AdaECE as the performance metrics and grid search does not require a differentiable objective function, we directly minimise ECE over the validation set during grid search.
Performance Gains: We report the optimal temperatures and their corresponding ECE% and AdaECE% (computed using 15 bins) in Tables 1 and 2 Table 3: Test set ECE and BLEU score both pre and post temperature scaling for cross-entropy (CE) with hard targets, cross-entropy with label smoothing (LS) (α = 0.1) and focal loss (FL) (γ = 1).
cross-entropy, label smoothing, Brier score and MMCE. It broadly produces the lowest ECE and AdaECE values both before and after temperature scaling. This observation is particularly encouraging, as it indicates that a principled method of obtaining values of γ for focal loss can work well. Furthermore, Tables 5 and 6 in the appendix show that other focal loss based approaches are also very competitive. Finally, we observe that there are cases where ECE might be low, implying that the model is well calibrated, whereas AdaECE evaluated on the same model might be high. For example, in the case of WideResNet on CIFAR-10 for cross-entropy, the best ECE obtained after temperature scaling is 0.92, whereas AdaECE on the same model at the same temperature is 1.69.

Confident and Calibrated Models:
It is interesting to note that for focal loss with sample-based γ (see Tables  1 and 2), the optimal temperatures are very close to 1, mostly lying between 0.9 and 1.1. This property is shown by the Brier score and the label smoothing models as well. By contrast, the optimal temperatures for the baselines (cross-entropy with hard targets and MMCE) are significantly higher, with values lying between 2.0 to 2.8. An optimal temperature close to 1 indicates that the model is innately calibrated and cannot be made significantly more calibrated by temperature scaling. Furthermore, an optimal temperature that is much greater than 1 can make the network underconfident in general, as its outputs are temperature-scaled irrespective of their correctness. We provide additional experimental and qualitative results to support this claim in Appendix F.

Machine Translation: A Downstream Task
In order to observe the performance of focal loss for a downstream task, where a calibrated model can potentially improve the performance on the task, we conduct an experiment on machine translation with beam search. Following the setup described in the work Müller et al. [2019], we train the Transformer architecture [Vaswani et al., 2017] on the standard WMT 2014 English-to-German translation dataset. The settings used for training (like optimiser, learning rate schedule, number of training iterations, etc.) are exactly the same as mentioned in the paper [Vaswani et al., 2017]. The intuition behind having machine translation as the downstream task of choice lies in the fact that in translation, the softmax vectors produced by the transformer model are directly fed into the beam search algorithm, and hence softmax vectors from a more calibrated model should ideally produce better translations and a better BLEU score.
We train three transformer models, one on cross-entropy with hard target labels, the second on cross-entropy with label smoothing (with smoothing factor α = 0.1) and the third on focal loss with γ = 1. In order to compare these models in terms of calibration, we report the test set ECE (%) both before and after temperature scaling in the first row of Table 3. Furthermore, to evaluate their performance on the English-to-German translation task, we also report the test set BLEU score of these models in the second row of Table 3. Finally, to study the variation of test set BLEU score and validation set ECE with temperature, we plot them against temperature for all three models in Figure 5.
It is clear from Table 3 that the model trained on focal loss outperforms its competitors on both ECE and BLEU score. The focal loss model also has an optimal temperature of 1, just like the model trained on cross-entropy with label smoothing. From Figure 5, we can see that the models obtain the highest BLEU scores at around the same temperatures at which they obtain low ECEs, thereby confirming our initial notion that a more calibrated model provides better translations. However, since the optimal temperatures are tuned on the validation set, they don't often correspond to the best BLEU scores on the test set. On the test set, the highest BLEU scores we observe are 26.33 for cross-entropy, 26.36 for cross-entropy with label smoothing, and 26.39 for focal loss. Thus, focal loss obtains both the lowest ECE and the highest BLEU.

Conclusion
In this paper, we have shown that training using focal loss can yield multi-class classification networks that are more naturally calibrated than those trained using the more conventional cross-entropy loss. There are sound theoretical reasons to expect this: in particular, as we show in §4, focal loss implicitly maximises entropy while minimising the KL divergence between the predicted and the target distributions. We also show that, because of its design, it naturally regularises the weights of a network during training, reducing NLL overfitting and thereby improving calibration. Extensive experiments on a variety of computer vision (CIFAR-10/100/Tiny-ImageNet) and NLP (20 Newsgroups/SST) datasets, with a wide variety of different network architectures, show that this expectation is also borne out in practice. Our results show that in almost all cases, networks trained with focal loss are more calibrated than those trained with cross-entropy loss, label smoothing, Brier score and MMCE, whilst having similar levels of accuracy, making their predictions much easier for downstream components to trust. Finally, we verify this by showing the superior performance of focal loss on the downstream task of English-to-German translation.

Acknowledgements
This work was started whilst J Mukhoti was at FiveAI, and completed after he moved to the University of Oxford. V Kulharia is wholly funded by a Toyota Research Institute grant. A Sanyal acknowledges support from The Alan Turing Institute under the Turing Doctoral Studentship grant TU/C/000023. This work was also supported by ERC grant ERC-2012-AdG 321162-HELIOS, EPSRC grant Seebibyte EP/M013774/1, EPSRC/MURI grant EP/N019474/1, and the Royal Academy of Engineering.

Appendix
In § A, we discuss the relation between focal loss and a regularised KL divergence where the regulariser is the entropy of the predicted distribution. In § B, we discuss the regularisation effect of focal loss on a simple setup, i.e., a generalised linear model trained on a simple data distribution. In § C, we show the proofs of the two propositions formulated in the main text. We then describe all the datasets and implementation details for our experiments in § D. In § E, we discuss additional approaches for training using focal loss and also the results we get from these approaches. We also provide Top-5 accuracies of several models to possibly hint at their calibration. We further provide results on evaluating our models using various metrics other than ECE and Ada-ECE (like MCE and NLL). In § F, we provide empirical and qualitative results to show that models trained using focal loss are calibrated while maintaining their confidence on correct predictions. Finally, in § G, we provide a brief extension on our discussion about Figure 3(e) in the main paper with a plot of L 2 norms of features obtained from the last ResNet block during training.

A Relation between Focal Loss and Entropy Regularised KL Divergence
Here we show why focal loss favours accurate but relatively less confident solutions. We show that it inherently provides a trade-off between minimizing the KL-divergence and maximizing the entropy, depending on the strength of γ. We use L f and L c to denote the focal loss with parameter γ and cross entropy betweenp and q, respectively. K denotes the number of classes and q y denotes the ground-truth probability assigned to the y-th class (similarly forp y ). We consider the following simple extension of focal loss: (1 − γp y )q y logp y By Bernoulli's inequality ∀γ ≥ 1. Note,p y ∈ [0, 1] We know that L c = KL(q||p) + H[q], thus, combining this equality with the above inequality leads to: In the case of one-hot encoding (Delta distribution for q), focal loss would maximize −p y logp y (let y be the ground-truth class index), the component of the entropy ofp corresponding to the ground-truth index. Thus, would prefer learningp such thatp y is assigned a higher value (because of the KL term) but not too high (because of the entropy term) which eventually would avoid preferring overconfident models (as opposed to the cross-entropy loss). Experimentally, we found the solution of the cross entropy and focal loss equations, i.e. the value of the predicted probabilityp which minimizes the loss, for various values of q in a binary classification problem (i.e. K = 2) and plotted it in Figure 6. As expected, focal loss favours a more entropic solutionp that is closer to 0.5. In other words, as Figure 6 shows, solutions to focal loss (Eqn 5) will always have higher entropy than that of cross entropy depending on the value of γ. The behaviour of deep neural networks is generally quite different from linear models and the problem of calibration is more pronounced in the case of deep neural networks, hence we focus on analysing the calibration of deep networks in the paper. However, to understand the effect of focal loss on a simpler setup, we also conducted experiments on a generalised linear model using a simple data distribution.
Setup We consider a binary classification problem. The data matrix X ∈ R 2×N is created by assigning each class, two normally distributed clusters such that the mean of the clusters are linearly separable. The mean of the clusters are situated on the vertices of a two-dimensional hypercube of side length 4. The standard deviation for each cluster is 1 and the samples are randomly linearly combined within each cluster in order to add covariance. Further, for 10% of the data points, the labels were flipped. 4000 samples are used for training and 1000 samples are used for testing. The model consists of a simple 2-parameter logistic regression model.
. We train this model using both cross-entropy and focal loss with γ = 1.

Weight Magnification
We have argued that focal loss implicitly regularizes the weights of the model by providing smaller gradients as compared to cross-entropy. This helps in calibration as, if all the weights are large, the logits are large and thus the confidence of the network is large on all test points, even on the misclassified points. When the model misclassifies, it misclassifies with a high confidence. Figure 7 shows, for a generalised linear model, that the norm of the logits and the weights of a network blows for Cross Entropy as compared to Focal Loss. Figures 8 (b) and (c) show that running gradient descent with cross-entropy (CE) and focal loss (FL) both gives the same decision regions i.e. the weight vector points in the same region for both FL and CE. However, as we have seen that the norm of the weights is much larger for CE as compared to FL, we would expect the confidence of misclassified test points to be large for CE as compared to FL. Figure 8 (a) plots a histogram of the confidence of the misclassified points and it shows that CE misclassifies almost always with greater than 90% confidence whereas FL misclassifies with much lower confidence.

C Proofs
Here we provide the proofs of both the propositions presented in the main text. While Proposition 1 helps us understand the regularization effect of focal loss, Proposition 2 provides us the γ values in a principled way such that it is sample-dependent. Implementing the sample-dependent γ is very easy as implementation of the Lambert-W function [Corless et al., 1996] is available in standard libraries (e.g. python scipy).
Proof. We derive the value of γ > 0 for which g(p 0 , γ) = 1 for a given p 0 ∈ [0, 1]. From Proposition 4.1, we already know that where L f is focal loss, L c is cross entropy loss,p i,yi is the probability assigned by the model to the ground-truth correct class for the i th sample, and For p ∈ [0, 1], if we look at the function g(p, γ), then we can clearly see that g(p, γ) → 1 as p → 0, and that g(p, γ) = 0 when p = 1. To observe the behaviour of g(p, γ) for intermediate values of p, we first take its derivative with respect to p: In Equation 8, γ(1 − p) γ−2 > 0 except when p = 1 (in which case γ(1 − p) γ−2 = 0). Thus, to observe the sign of the gradient ∂g(p,γ) ∂p , we focus on the term Dividing Equation 9 by (− log p), the sign remains unchanged and we get We can see that k(p, γ) → 1 as p → 0 and k(p, γ) → −(1 + γ) as p → 1 (using l'Hôpital's rule). Furthermore, k(p, γ) is monotonically decreasing for p ∈ [0, 1]. Thus, as the gradient ∂g(p,γ) ∂p monotonically decreases from a positive value at p = 0 to a negative value at p = 1, we can say that g(p, γ) first monotonically increases starting from 1 (as p → 0) and then monotonically decreases down to 0 (at p = 1). Thus, if for some threshold p 0 > 0 and for some γ > 0, g(p, γ) = 1, then ∀p > p 0 , g(p, γ) < 1. We now want to find a γ such that ∀p ≥ p 0 , g(p, γ) ≤ 1. First, let a = (1 − p 0 ) and b = p 0 log p 0 . Then: where a = (1 − p 0 ) and b = p 0 log p 0 . We know that the inverse of y = xe x is defined as x = W (y), where W is the Lambert-W function [Corless et al., 1996]. Furthermore, the r.h.s. of the inequality in Equation 11 is always negative, with a minimum possible value of −1/e that occurs at p 0 = 0.5. Therefore, applying the Lambert-W function to the r.h.s. will yield two real solutions (corresponding to a principal branch denoted by W 0 and a negative branch denoted by W −1 ). We first consider the solution corresponding to the negative branch (which is the smaller of the two solutions): If we consider the principal branch, the solution is which yields a negative value for γ that we discard. Thus Equation 12 gives the values of γ for which if p > p 0 , then g(p, γ) < 1. In other words, g(p 0 , γ) = 1, and for any p < p 0 , g(p, γ) > 1.

D Dataset Description and Implementation Details
We use the following image and document classification datasets in our experiments: 1. CIFAR-10 [Krizhevsky, 2009]: This dataset has 60,000 colour images of size 32 × 32, divided equally into 10 classes. We use a train/validation/test split of 45,000/5,000/10,000 images. [Socher et al., 2013]: This dataset contains movie reviews in the form of sentence parse trees, where each node is annotated by sentiment. We use the dataset version with binary labels, for which 6,920/872/1,821 documents are used as the training/validation/test split. In the training set, each node of a parse tree is annotated as positive, neutral or negative. At test time, the evaluation is done based on the model classification at the root node, i.e. considering the whole sentence, which contains only positive or negative sentiment.

Stanford Sentiment Treebank (SST)
For training networks on CIFAR-10 and CIFAR-100, we use SGD with a momentum of 0.9 as our optimiser, and train the networks for 350 epochs, with a learning rate of 0.1 for the first 150 epochs, 0.01 for the next 100 epochs, and 0.001 for the last 100 epochs. We use a training batch size of 128. Furthermore, we augment the training images by applying random crops and random horizontal flips. For Tiny-ImageNet, we train for 100 epochs with a learning rate of 0.1 for the first 40 epochs, 0.01 for the next 20 epochs and 0.001 for the last 40 epochs. We use a training batch size of 64. It should be noted that for Tiny-ImageNet, we saved 50 samples per class (i.e., a total of 10000 samples) from the training set as our own validation set to fine-tune the temperature parameter (hence, we trained on 90000 images) and we use the Tiny-ImageNet validation set as our test set.
For 20 Newsgroups, we train the Global Pooling Convolutional Network [Lin et al., 2014] using the Adam optimiser, with learning rate 0.001, and betas 0.9 and 0.999. The code is a PyTorch adaptation of Ng. We used Glove word embeddings [Pennington et al., 2014] to train the network. We trained all the models for 50 epochs and used the models with the best validation accuracy.
For the SST Binary dataset, we train the Tree-LSTM [Tai et al., 2015] using the AdaGrad optimiser with a learning rate of 0.05 and a weight decay of 10 −4 , as suggested by the authors. We used the constituency model, which considers binary parse trees of the data and trains a binary tree LSTM on them. The Glove word embeddings [Pennington et al., 2014] were also tuned for best results. The code framework we used is inspired by TreeLSTM. We trained these models for 25 epochs and used the models with the best validation accuracy.
For all our models, we use the PyTorch framework, setting any hyperparameters not explicitly mentioned to the default values used in the standard models. For MMCE, we used λ = 2 for all the image-classification tasks, whilst we found λ = 8 to perform better for document classification. A calibrated model which does not generalise well to an unseen test set is not very useful. Hence, for all the experiments, we set the training parameters in a way such that we get state-of-the-art test set accuracies on all datasets for each model.

E Additional Results
In addition to the sample-dependent γ approach, we try the following focal loss approaches as well: Focal Loss (Fixed γ): We trained models on focal loss with γ fixed to 1, 2 and 3. We found γ = 3 to produce the best ECE among models trained using a fixed γ. This corroborates the observation we made in §4 of the main paper that γ = 3 should produce better results than γ = 1 or γ = 2, as the regularising effect for γ = 3 is higher.
Focal Loss (Scheduled γ): As a simplification to the sample-dependent γ approach, we also tried using a schedule for γ during training, as we expect the value ofp i,yi to increase in general for all samples over time. In particular, we report results for two different schedules: (a) Focal Loss (scheduled γ 5,3,2): γ = 5 for the first 100 epochs, γ = 3 for the next 150 epochs, and γ = 2 for the last 100 epochs, and (b) Focal Loss (scheduled γ 5,3,1): γ = 5 for the first 100 epochs, γ = 3 for the next 150 epochs, and γ = 1 for the last 100 epochs. We also tried various other schedules, but found these two to produce the best results on the validation sets.
In Table 4 we present the classification errors on the test datasets for all the major loss functions we considered. Moreover, we also report the classification errors for the different focal loss approaches in Table 7. We also report the ECE and Ada-ECE for all the focal loss approaches in Table 5 and Table 6.
Finally, calibrated models should have a higher logit score (or softmax probability) on the correct class even when they misclassify, as compared to models which are less calibrated. Thus, intuitively, such models should have a higher Top-5 accuracy. In Table 8, we report the Top-5 accuracies for all our models on datasets where the number of classes is relatively high (i.e., on CIFAR-100 with 100 classes and Tiny-ImageNet with 200 classes). We observe focal loss with sample-dependent γ to produce the highest top-5 accuracies on all models trained on CIFAR-100 and the second best top-5 accuracy (only marginally below the highest accuracy) on Tiny-ImageNet.
In addition to ECE and Ada-ECE, we use various other metrics to compare the proposed methods with the baselines (i.e. cross-entropy, Brier score, MMCE and Label Smoothing). We present the test NLL % before and after temperature scaling in Tables 9 and 10, respectively. We report the test set MCE % before and after temperature scaling in Tables 11 and 12, respectively.

F Focal Loss is Confident and Calibrated
In extension to what we present in Section 5 of the main paper, we also follow the approach adopted in Kumar et al. [2018], and measure the percentage of test samples that are predicted with a confidence of 0.99 or more (we call this set of test samples S99). In Table 13 Table 9: NLL (%) computed for different approaches pre temperature scaling.
using cross-entropy loss, MMCE loss, and focal loss. We observe that |S99| for the focal loss model is much lower than for the cross-entropy or MMCE models before temperature scaling. However, after temperature scaling, |S99| for focal loss is significantly higher than for both MMCE and cross-entropy. The reason is that with an optimal temperature of 1.1, the confidence of the temperature-scaled model for focal loss does not reduce as much as those of the models for cross-entropy and MMCE, for which the optimal temperatures lie between 2.5 to 2.8. We thus conclude that models trained on focal loss are not only more calibrated, but also better preserve their confidence on predictions, even after being post-processed with temperature scaling.
In Figure 9, we present some qualitative results to support this claim and show the improvement in the confidence estimates of focal loss in comparison to other baselines (i.e., cross entropy, MMCE and Brier score). For this, we    take ResNet-50 networks trained on CIFAR-10 using the four loss functions (cross entropy, MMCE, Brier score and Focal loss with sample-dependent γ 5,3) and measure the confidence of their predictions for four correctly and four incorrectly classified test samples. We report these confidences both before and after temperature scaling. It is clear from Figure 9 that for all the correctly classified samples, the model trained using focal loss has very confident predictions both pre and post temperature scaling. However, on misclassified samples, we observe a very low confidence for the focal loss model. The ResNet-50 network trained using cross entropy is very confident even on the misclassified samples, particularly before temperature scaling. Apart from focal loss, the only model which has relatively low confidences on misclassified test samples is the one trained using Brier score. These observations support our claim that focal loss produces not only a calibrated model but also one which is confident on its correct predictions.

G Ordering of Feature Norms
As an extension to the discussion related to Figure 3(e) in the main paper, we plot the L 2 norm of the features/activations obtained from the last ResNet block (right before the linear layer is applied on these features to get the logits). We plot these norms throughout the training period for networks trained on cross-entropy and focal loss with γ set to 1, 2 and 3 in Figure 10. We observe that there is a distinct ordering of feature norms for the four models: cross-entropy has the highest feature norm, followed by focal loss with γ = 1, followed by focal loss with γ = 2 and finally focal loss with γ = 3. Furthermore, this ordering is preserved throughout training. As we saw from Figure 3(e) in the main paper, from epoch 150 onwards (i.e., the epoch from which the networks start getting miscalibrated), there is a flip in the ordering of weight norms of the last linear layer. From epoch 150 onwards, the weight norms also follow the exact same ordering that we observe from Figure 10 here. This shows that throughout training the initial layer weights (before last linear layer) of the network trained using focal loss are also regularized to favor lower norm of the output features, thus possibly leading to less peakiness in final prediction as compared to that of cross-entropy loss (see the 'Peak at the wrong place' paragraph of Section 3 of the main paper).