{'fraction_non_alphanumeric': 0.03730533986899753, 'fraction_numerical': 0.021428881273586952, 'mean_word_length': 4.583434245580044, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Margin Calibration for Long-Tailed Visual Recognition

The long-tailed class distribution in visual recognition tasks poses great challenges for neural networks on how to handle the biased predictions between head and tail classes, i.e., the model tends to classify tail classes as head classes. While existing research focused on data resampling and loss function engineering, in this paper, we take a different perspective: the classification margins. We study the relationship between the margins and logits (classification scores) and empirically observe the biased margins and the biased logits are positively correlated. We propose MARC, a simple yet effective MARgin Calibration function to dynamically calibrate the biased margins for unbiased logits. We validate MARC through extensive experiments on common long-tailed benchmarks including CIFAR-LT, ImageNet-LT, Places-LT, and iNaturalist-LT. Experimental results demonstrate that our MARC achieves favorable results on these benchmarks. In addition, MARC is extremely easy to implement with just three lines of code. We hope this simple method will motivate people to rethink the biased margins and biased logits in long-tailed visual recognition.


Introduction
Despite the great success of neural networks in the visual recognition field (Simonyan and Zisserman 2014;He et al. 2016), it is still challenging for neural networks to deal with the ubiquitous long-tailed datasets in the real world (Buda, Maki, and Mazurowski 2018;Kang et al. 2019;Zhou et al. 2020). To be clear, in the long-tailed datasets, the highfrequency classes (head classes) occupy most of the instances, whereas the low-frequency classes (tail classes) involve a small amount of instances Van Horn and Perona 2017). Due to the imbalance of the training data, the model performs well in head classes and its performance is much worse in tail classes (Buda, Maki, and Mazurowski 2018;Zhang et al. 2021).
Towards addressing the long-tailed recognition problem, there are several strategies such as data re-sampling and loss function engineering. Data re-sampling aims to 'simulate' a balanced training dataset by over-sampling the tail class or under-sampling the head classes (Ando and Huang 2017; * These authors contributed equally. † Corresponding author. Preprint. Buda, Maki, and Mazurowski 2018; Pouyanfar et al. 2018;Shen, Lin, and Huang 2016), while loss re-weighting is introduced to adjust the weights of losses for different classes or different instances (Byrd and Lipton 2019;Khan et al. 2017;Wang, Ramanan, and Hebert 2017). For more balanced gradients between classes, some class-balanced loss functions adjust the logits(classification scores) instead of weighting the losses (Menon et al. 2020;Cao et al. 2019a;Ren et al. 2020). However, as pointed out by existing research (Ganganwar 2012; Zhou and Liu 2005;Cao et al. 2019a), data resampling strategies and loss re-weighting schemes will possibly cause underfitting on the head class and overfitting on the tail class. On the other hand, class-balanced loss functions or data re-sampling will lead to worse data representations compared with the standard training using the crossentropy loss and the instance-balanced sampling (i.e., each instance has the same probability of being sampled) (Kang et al. 2019;Ren et al. 2020). In addition, the recent research reveals that the biased decision boundary given by the classifier head seems to be the performance bottleneck of the long-tailed visual recognition (Kang et al. 2019;Zhang et al. 2021). To benefit from both good data representations and the unbiased decision boundary, Decoupling, a heuristic two-stage strategy is proposed to adjust the initially-learned classifier head (Kang et al. 2019) after the standard training. Furthermore, distribution alignment (Zhang et al. 2021) is developed as an adaptive calibration function to adjust the initially trained classification scores for each data point. However, the relation between the biased margin and the biased logit is neglected in existing research, where the margin is the distance from the data point to the decision boundary.
In this paper, we study the relationship between margins and logits, which are the fundamental factors that dominate the long-tailed performance. We empirically find that the margin and the logit are correlated with the cardinality of each class. To be concrete, as shown in Figure 1, head classes tend to have much larger margins and logits than tail classes. Hence, to obtain the unbiased logit, calibrating the biased margin is necessary. Inspired by these evidence, we propose a simple yet effective MARgin Calibration (MARC) for long-tailed recognition. In detail, after getting the representations and the classifier head from the standard training, we propose a simple class-specific margin calibration function with only 2K learnable parameters to dynamically adjust the initially learned margins, where K is the number of classes. As demonstrated in Figure 1, the logits are more balanced when using MARC.
We conduct experiments on CIFAR-10-LT (Krizhevsky, Hinton et al. 2009), CIFAR-100-LT (Krizhevsky, Hinton et al. 2009), Places-LT (Zhou et al. 2017), iNaturalist 2018(Van Horn et al. 2018 and ImageNet-LT . The results illustrate that our proposed MARC method performs remarkably well even if MARC is very simple to implement. We hope our exploration will attract attention to the biased margins in long-tailed recognition.
To conclude, our contributions are as follows: • For the first time, we study the biased predictions from a margin-based perspective in long-tailed recognition. We find that biased margins will cause biased predictions. • Based on our observations, we propose a simple yet effective margin calibration (MARC) function with only 2K trainable parameters to adjust the margin and get the unbiased prediction. • MARC performs notably well on various long-tailed visual benchmarks like CIFAR-10-LT (2009), CIFAR-100-LT (2009), ImageNet-LT (2019), Places-LT (2017) and iNaturalist2018 (2018). In addition, it is extremely easy to implement with just three lines of code.

Related Work
Long-tailed visual recognition has attracted much attention for its commonness in the real world (He and Garcia 2009;Buda, Maki, and Mazurowski 2018;Kang et al. 2019;Ren et al. 2020;Yang and Xu 2020;Hong et al. 2021). Existing methods can be divided into four categories. Data re-sampling. Data re-sampling technologies resample the imbalanced training dataset to 'simulate' a balanced training dataset. These methods include undersampling, over-sampling, and classed-balanced sampling. Under-sampling decreases the probability of the instance of head classes being sampled (Drummond, Holte et al. 2003), whereas over-sampling makes instances of tail classes more likely to be sampled (Chawla et al. 2002;Han, Wang, and Mao 2005;Wang et al. 2021a). Class-aware sampling chooses instances of each class with the same probabilities (Shen, Lin, and Huang 2016).
Loss function engineering. Loss function engineering is another direction to obtain balanced gradients during the training. The typical methods can be categorized as lossreweighting and logits adjustment. Loss re-weighting adjusts the weights of losses for different classes or different instances in a more balanced manner, i.e. the instances in tail classes have larger weights than those in head classes (Byrd and Lipton 2019;Khan et al. 2017;Wang, Ramanan, and Hebert 2017). On the other hand, instead of re-weight losses, some class-balanced loss functions adjust the logits to get balanced gradients during training (Menon et al. 2020;Cao et al. 2019a;Ren et al. 2020;Yang, Yang, and Wang 2009).
Decision boundary adjustment. Nevertheless, data resampling or loss function engineering will influence the representations of data (Ren et al. 2020). Lots of empirical observations show that we can acquire good representation when using the standard training and the classifier head is the performance bottleneck (Kang et al. 2019;Zhang et al. 2021;Yu et al. 2020;Kim and Kim 2020). To solve the above problem, decision boundary adjustment methods re-adjust the classifier head after the standard training (Kang et al. 2019;Zhang et al. 2021). However, they ignore the relationship between the biased margins and the biased logits.
Other methods. There also exist other paradigms to deal with the long-tailed recognition task, including task-specific neuron networks design (Wang et al. 2021c;Zhou et al. 2020;Wang et al. 2021a), transfer learning Yin et al. 2019), domain adaptation (Jamal et al. 2020), semi supervised learning and self supervised learning (Yang and Xu 2020). But these methods either rely on the non-trivial network design or external data. In contrast, our proposed MARC is very simple to implement and does not require external data. The detailed comparison between MARC and similar methods is shown in Table 1.

Method Preliminaries
In the popular setting of long-tailed recognition (Kang et al. 2019;Cui et al. 2019;Ren et al. 2020), the training data distribution is imbalanced while the test data distribution is balanced. More formally, let D = {(x i , y i )} n i=1 be a training set, where y i denotes the label of data point x i . Specifically, n = K j=1 n j is the total number of training samples, where n j is the number of training samples in class j and K is the number of classes. We assume n 1 > n 2 > · · · > n K without loss of generality. Normally, the prediction function is composed of two modules: the feature representation learning function f : x → z parameterized by θ r and the classifier g : z → y parameterized by θ c , where z ∈ R p denotes the feature representation and p is the feature dimension. Typically, g is a linear classifier that gives the classification score of class j as: η j = g(z) := W j z + b j with W j and b j its weight vector and bias for class j. Finally, using the softmax function, the probability of x i being classified as label y i is expressed as: and its loss is computed as the cross-entropy loss: (2)

Motivation
The decision boundary is often biased in long-tailed recognition, which will lead to biased predictions, i.e., the model tends to classify tail classes as head classes. To alleviate this issue, data re-sampling and loss function engineering are two directions to simulate a 'balanced' training dataset. However, such techniques will do harm to the representation learning of the model (Kang et al. 2019;Ren et al. 2020). To benefit from both the good representation and the unbiased decision boundary, decision boundary adjusts methods are developed (Kang et al. 2019;Zhang et al. 2021). However, existing decision boundary adjusts methods ignore the bias in the margins, which we think is essential to avoid biased predictions. Thus in this paper, we aim to calibrate the biased margins to obtain unbiased predictions.

Biased Margins and Biased Logits
In this paper, we find that the margins and logits are biased in long-tailed recognition. We illustrate margins with Figure 2. We define an affine hyperplane H j ∈ R p−1 of class j as W j z + b j = 0, i.e. any representation point falling on the positive side of H j can be attributed to class j. Assume that z 0 is a point satisfying W j z 0 + b j = 0, i.e., z 0 is a point on the hyperplane H j . Suppose z 1 is an arbitrary point in the feature space. We construct the vector z 1 − z 0 pointing from z 0 to z 1 and project it onto the normal vector W j . The length of the projection vector proj Wj (z 1 − z 0 ) is the margin from z 1 to H j . Specifically, the margin from z 1 to the hyperplane of class j is calculated as: where · denotes L2 norm. Thus, the logit W j · z 1 + b j can also be expressed as W j d j . Based to this conclusion, we can rewrite Eq. (1) as: We assume a data point is on the decision boundary of class j and class t, i.e., such data point has the same probability of being classified as class j or class t. Clearly, the assumed data point on the decision boundary satisfies Figure 2: Illustration of margins. The red and blue dots denote majority and minority classes, respectively.
According to Eq. (5), data will be classified as class t because d j < d t when W j = W t . And our empirical observations show that head classes tend to have much larger margins and logits than tail classes: whered j is the average margin of class j andη j is the average logit of class j after the standard training. In detail,

Margin Calibration Function (MARC)
To get the unbiased logit, we propose to calibrate the margins after the standard training. Concretely, we train a simple class-specific margin calibration model with the original margin fixed:d where ω j and β j are learnable parameters for class j and j ∈ [1, K]. In other words, MARC only has 2K trainable parameters. Then the calibrated logit is computed as: where η j is the initial fixed logit. Then we can get the calibrated prediction distribution: The training process of the margin calibration function can be written with just three lines of Pytorch codes as shown in Line 4-6 of Algorithm 1. Furthermore, for more balanced gradients during training, we re-weight the loss as the previous work does (Zhang et al. 2021). Finally, the loss for training the margin calibration function is: logit before = model(x) 6: logit after = omega * logit before \ + beta * w norm 7: Compute loss and update parameters of omega and beta.
where parameters with· are frozen during training and the weight for class y i is calculated as: where γ is a scale hyper-parameter. When γ = 0, the weight for all classes is 1, which means no re-weighting at all.
To be more clear, the whole detailed training procedure including both standard training and margin calibration function training is demonstrated in Algorithm 2. Lines 2-6 include the training procedure of the standard training using the instance-balanced sampling and the cross-entropy loss. Lines 7-11 contain the training process of our margin calibration function. It is well noting that in the second stage, parameters θ r and θ c are all fixed.

Discussion
We clarify the differences between MARC and other learnable decision boundary adjustment methods in detail. As shown in Table 1, Decouple-cRT (Kang et al. 2019) retrains the whole parameters of the classifier, while Decouple-LWS (Kang et al. 2019) only adjusts the norm of weight vectors W j . Instead of adjusting the classifier head, Dis-Align (Zhang et al. 2021) chooses to calibrate the logit for each data point. But their calibration method is heuristic that simply adds the calibrated logit and the original logit with a re-weighting scheme. To be more clear, the weighted sum of logits for DisAlign is σ(z j )(ω j η j + β) + (1 − σ(z j ))η j , where σ(·) is an instance-specific confidence function. However, different from previous methods, our MARC focuses on calibrating the biased margin which we believe is the performance bottleneck of the long-tailed classifier.
Algorithm 2: The detailed training procedure including both standard training and margin calibration function training.
1: Input: The training dataset D = {(x i , y i )} n i=1 , the parameters of the representation function θ r , the parameters of the classifier θ c , the parameters of the margin calibration function ω and β, the number of classes K and the pre-defined scale hyper-parameter γ. Compute the loss and update the model parameters.

Experiments
In this section, we conduct extensive experiments compared with the state-of-the-art methods to validate the effectiveness of MARC. Firstly, we report the performance on common benchmarks like CIFAR-10-LT (2009) (2018). The imbalance factor used in CIFAR datasets is defined as N max /N min where N max is the number of samples on the largest class and N min the smallest. We report CIFAR results with two different imbalance ratios: 100 and 200. For ImageNet-LT and Places-LT experiments, we further split classes into three sets: Many-shot (with more than 100 images), Medium-shot (with 20 to 100 images), and Few-shot (with less than 20 images).
Training Configuration For a fair comparison, our experiments are conducted under the most commonly used For CIFAR-10-LT and CIFAR-100-LT, the models are trained for 13,000 iterations with a batch size of 512. We use the SGD optimizer with momentum 0.9 and weight decay 5e − 4 for all datasets except for iNatu-ralist18 where the weight decay is 1e − 4. In the standard training stage, we use a cosine learning rate schedule with an initial value of 0.05 for CIFAR and 0.1 for other datasets, which gradually decays to 0. In the margin calibration stage, we use a cosine learning rate schedule with an initial learning rate starting from 0.05 to 0 for all datasets. γ is set to 1.2 for all datasets. The hyper-parameters of compared methods follow their paper. For fairness, we use the same pre-trained model for decision boundary adjustment methods.

Comparison with previous methods
In this section, we compare the performance of MARC to other recent works. We select some recent methods from each of the following four categories for comparison: data re-sampling, loss function engineering, decision boundary adjustment, others. The standard training with the crossentropy loss and instance balance sampling is called Softmax in our results.

ImageNet-LT
We further evaluate MARC on the ImageNet-LT dataset. As Table 3 shows, MARC is better than all loss function engineering methods. Compared with LADE, although our overall accuracy is 0.4% higher, the accuracy on the few-shot classes is 5.4% higher. The  few-shot accuracy and overall accuracy of MARC are 1.9% and 0.1% higher than DisAlign respectively. Our results are quite surprising considering the simplicity of MARC.
Places-LT For the Places-LT dataset, MARC achieves better performance than other decision boundary adjustment methods. Though our overall accuracy is lower than LADE, our few-shot accuracy is still 1.4% higher than LADE. Though MARC is not the best in Places-LT, the results of MARC are still competitive compared with other methods.
iNaturalist-LT Finally, we present the Top-1 accuracy results for iNaturalist-LT dataset in Table 5. We can observe a similar trend that our proposed method wins all the existing approaches and surpasses DisAlign by 0.1% absolute improvement.

Further analysis
In this section, we conduct different experiments for further analysis. To be concrete, we empirically show that MARC can achieve more balanced margins and logits compared with DisAlign. Moreover, the class-wise accuracy of MARC is much better than the standard training baseline model on  CIFAR-LT, which indicates that we can alleviate the biased prediction problem and reduce the performance gap between the head classes and the tail classes.
Visualization of the margin and logit In this subsection, we visualize the values of margins and logits for each class to show the effect of MARC. As illustrated both in Figure 3, Figure 5c and Figure 5d, before margin calibration, the margins and the logits are biased, i.e. the head classes tend to have much larger margins and logits than tail classes. We believe the bias in margins and logits will lead to biased predictions in long-tailed visual recognition. The biased margins and logits become more balanced after the margin calibration. This result proves that we can get more unbiased predictions by calibrating the margin. Moreover, as shown in Figure 5a and Figure 5b, MARC will obtain more balanced margins and gradients than DisAlign. The instability of DisAlign may be caused by their heuristic design of the combination of the calibrated logits and the origin logits.
Class-wise performance on CIFAR-LT As we can see in Figure 4, after our margin calibration method, the performance on tail classes is improved while that on head classes is not severely affected. More intuitively, Figure 6 shows the class-wise performance. The accuracy of the tail class is much higher than that of the head class. The performance degradation on head classes may be caused by the false positive predictions on head classes, i.e., the standard training method tends to classify tail classes as head classes, resulting in high accuracy on the head classes. The bad performance on tail classes when using standard training also proves this. In addition, the overall accuracy and F1-score show that MARC alleviates the biased prediction problem.    Effects of different γ To explore the effect of different γ, we also conduct experiments and visualize the performances on all CIFAR-100-LT(200). The results are shown in Figure 7b. We can observe that compared 1.2 is the best compared with other values. γ can not be too large since in this way the weight for head classes is too small. It is well noting that MARC also achieves 45.2% accuracy when γ is 0. This means MARC still works even if we do not use any loss re-weighting techniques in the second stage. For other datasets, we directly use 1.2 for γ.
Effects of different standard pre-trained model We use different standard pre-trained models on CIFAR-100-LT(200) to explore their effects. As Table 6 illustrates, as the pre-trained dataset gets more balanced, the performance of our margin calibration method gets better. This shows that when comparing decision boundary methods, using the same standard training model is very important for fairness. And the codebase will also affect the standard training. This is the reason why the result of DisAlign in our paper is inconsistent with the original paper since we cannot get the same standard pre-trained model they used. So instead, we use our own standard pre-trained model for MARC and Dis-Align for fairness. Table 6 also demonstrates that the margin calibration method can achieve better performance when given better representations. So our future works include how to get better representations.
Comparison of the trainable parameters of decision boundary adjustment methods As shown in Table 7, MARC achieves the best performance among other compared methods on CIFAR-100-LT(200) and ImageNet-LT. Even though our trainable parameters are more than Decouple-LWS, our performance is better. Besides, it is surprising that MARC obtains such a favorable performance with only so few parameters.

Conclusions
This paper studied the long-tailed visual recognition problem. Specifically, we found that head classes tend to have much larger margins and logits than tail classes. Motivated by our findings, we proposed a margin calibration function with only 2K learnable parameters to obtain the unbiased logits in long-tailed visual recognition. Even though our method is very simple to implement, extensive experiments show that MARC achieves favorable results compared with previous methods. We hope that our study on logits and margins could provide experience for future research. In the future, we plan to explore better representations for long-tailed decision ad-