{'fraction_non_alphanumeric': 0.05407054516439417, 'fraction_numerical': 0.016615427941141958, 'mean_word_length': 4.307922272047833, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 3, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Rank&Sort Loss for Object Detection and Instance Segmentation

We propose Rank&Sort (RS) Loss, a ranking-based loss function to train deep object detection and instance segmentation methods (i.e. visual detectors). RS Loss supervises the classifier, a sub-network of these methods, to rank each positive above all negatives as well as to sort positives among themselves with respect to (wrt.) their localisation qualities (e.g. Intersection-over-Union - IoU). To tackle the non-differentiable nature of ranking and sorting, we reformulate the incorporation of error-driven update with backpropagation as Identity Update, which enables us to model our novel sorting error among positives. With RS Loss, we significantly simplify training: (i) Thanks to our sorting objective, the positives are prioritized by the classifier without an additional auxiliary head (e.g. for centerness, IoU, mask-IoU), (ii) due to its ranking-based nature, RS Loss is robust to class imbalance, and thus, no sampling heuristic is required, and (iii) we address the multi-task nature of visual detectors using tuning-free task-balancing coefficients. Using RS Loss, we train seven diverse visual detectors only by tuning the learning rate, and show that it consistently outperforms baselines: e.g. our RS Loss improves (i) Faster R-CNN by ~ 3 box AP and aLRP Loss (ranking-based baseline) by ~ 2 box AP on COCO dataset, (ii) Mask R-CNN with repeat factor sampling (RFS) by 3.5 mask AP (~ 7 AP for rare classes) on LVIS dataset; and also outperforms all counterparts. Code is available at: https://github.com/kemaloksuz/RankSortLoss


Introduction
Owing to their multi-task (e.g. classification, box regression, mask prediction) nature, object detection and instance segmentation methods rely on loss functions of the form: which combines L k t , the loss function for task t on stage k (e.g. |K| = 2 for Faster R-CNN [32] with RPN and R-* Equal contribution for senior authorship. Our RS Loss, in addition to raking positives above negatives, aims to sort positives wrt. their continuous IoUs (positives: a green tone based on its label, negatives: orange). We propose Identity Update (Section 3), a reformulation of error-driven update with backpropagation, to tackle these ranking and sorting operations which are difficult to optimize due to their non-differentiable nature. CNN), weighted by a hyper-parameter λ k t . In such formulations, the number of hyper-parameters can easily exceed 10 [28], with additional hyper-parameters arising from taskspecific imbalance problems [29], e.g. the positive-negative imbalance in the classification task, and if a cascaded architecture is used (e.g. HTC [7] employs 3 R-CNNs with different λ k t ). Thus, although such loss functions have led to unprecedented successes, they require tuning, which is time consuming, leads to sub-optimal solutions and makes fair comparison of methods challenging.
Recently proposed ranking-based loss functions, namely "Average Precision (AP) Loss" [6] and "average Localisation Recall Precision (aLRP) Loss" [28], offer two important advantages over the classical score-based functions (e.g. Cross-entropy Loss and Focal Loss [22]): (1) They directly optimize the performance measure (e.g. AP), thereby providing consistency between training and evaluation objectives. This also reduces the number of hyper-parameters as the performance measure (e.g. AP) does not typically have any hyper-parameters. (2) They are robust to class-imbalance due to their ranking-based error definition. Although these losses have yielded state-of-the-art (SOTA) performances, they need longer training and more augmentation.
Broadly speaking, the ranking-based losses (AP Loss and aLRP Loss) focus on ranking positive examples over negatives, but they do not explicitly model positive-topositive interactions. However, there is evidence that it is helpful to prioritize predictions wrt. their localisation qualities by using an auxiliary (aux. -e.g. IoU, centerness) head [15,17,38,44] or by supervising the classifier to directly regress IoUs of the predictions without an aux. head (as shown by Li et al. [18] in Quality Focal Loss -QFL).
In this paper, we propose Rank & Sort (RS) Loss as a ranking-based loss function to train visual detection (VD -i.e. object detection and instance segmentation) methods. RS Loss not only ranks positives above negatives ( Fig.  1(a)) but also sorts positives among themselves with respect to their continuous IoU values ( Fig. 1(b)). This approach brings in several crucial benefits. Due to the prioritization of positives during training, detectors trained with RS Loss do not need an aux. head, and due to its ranking-based nature, RS Loss can handle extremely imbalanced data (e.g. object detection [29]) without any sampling heuristics. Besides, except for the learning rate, RS Loss does not need any hyper-parameter tuning thanks to our tuning-free taskbalancing coefficients. Owing to this significant simplification of training, we can apply RS Loss to different methods (i.e. multi-stage, one-stage, anchor-based, anchor-free) easily (i.e. only by tuning the learning rate) and demonstrate that RS Loss consistently outperforms baselines.
Our contributions can be summarized as follows: (1) We reformulate the incorporation of error-driven optimization into backpropagation to optimize nondifferentiable ranking-based losses as Identity Update, which uniquely provides interpretable loss values during training and allows definition of intra-class errors (e.g. the sorting error among positives).
(2) We propose Rank & Sort Loss that defines a ranking objective between positives and negatives as well as a sorting objective to prioritize positives wrt. their continuous IoUs. Due to this ranking-based nature, RS Loss can train models in the presence of highly imbalanced data.
(3) We present the effectiveness of RS Loss on a diverse set of four object detectors and three instance segmentation methods only by tuning the learning rate and without any aux. heads or sampling heuristics on the widely-used COCO and long-tailed LVIS benchmarks: E.g. (i) Our RS-R-CNN improves Faster-CNN by ∼ 3 box AP on COCO, (ii) our RS-Mask R-CNN improves repeat factor sampling by ∼ 3.5 mask AP ( ∼ 7 AP for rare classes) on LVIS.

Related Work
Auxiliary heads and continuous labels. Predicting the localisation quality of a detection with an aux. centerness [38,44], IoU [15,17], mask IoU [14] or uncertainty head [13] and combining these predictions with the classification scores for NMS are shown to improve detection performance. Lin et al. [18] discovered that using continuous IoUs of predictions to supervise the classifier outperforms using an aux. head. Currently, Lin et al.'s "Quality Focal Loss" [18] is the only method that is robust to class imbalance [29] and uses continuous labels to train the classifier. With RS Loss, we investigate the generalizability of this idea on different networks (e.g. multi-stage networks [2,32]) and on a different task (i.e. instance segmentation).
Ranking-based losses in VD. Despite their advantages, ranking-based losses are non-differentiable and difficult to optimize. To address this challenge, black-box solvers [34] use an interpolated AP surface, though yielding little gain in object detection. DR Loss [31] achieves ranking between positives and negatives by enforcing a margin with Hinge Loss. Finally, AP Loss [6] and aLRP Loss [28] optimize the performance metrics, AP and LRP [26] respectively, by using the error-driven update of perceptron learning [35] for the non-differentiable parts. However, they need longer training and heavy augmentation. The main difference of RS Loss is that it also considers continuous localisation qualities as labels.
Objective imbalance in VD. The common strategy in VD is to use λ k t (Eq. 1), a scalar multiplier, on each task and tune them by grid search [1,17]. Recently, Oksuz et al. [28] employed a self-balancing strategy to balance classification and box regression heads, both of which compete for the bounded range of aLRP Loss. Similarly, Chen et al. [5] use the ratio of classification and regression losses to balance these tasks. In our design, each loss L k t for a specific head has its own bounded range and thus, no competition ensues among heads. Besides, we use L k t s with similar ranges, and show that our RS Loss can simply be combined with a simple task balancing strategy based on loss values, and hence does not require any tuning except the learning rate.

Identity Update for Ranking-based Losses
Using a ranking-based loss function is attractive thanks to its compatibility with common performance measures (e.g. AP). It is challenging, however, due to the nondifferentiable nature of ranking. Here, we first revisit an existing solution [6,28] that overcomes this nondifferentiability by incorporating error-driven update [35] into backpropagation (Section 3.1), and then present our reformulation (Section 3.2), which uniquely (i) provides interpretable loss values and (ii) takes into account intra-class errors, which is crucial for using continuous labels.

Logits
Difference Transforms Step 1 Step 2 Step 3 Figure 2. Three-step computation (green arrows) and optimization (orange arrows) algorithms of ranking-based loss functions. Our identity update (i) yields interpretable loss values (see Appendix A for an example on our RS Loss), (ii) replaces Eq. 2 of previous work [28] by Eq. 5 (green arrow in Step 2) to allow intra-class errors, crucial to model our RS Loss, and (iii) results in a simple "Identity Update" rule (orange arrow in Step 2): ∆xij = Lij.

Revisiting the Incorporation of Error-Driven Optimization into Backpropagation
Definition of the Loss. Oksuz et al. [28] propose writing a ranking-based loss as L = 1 Z i∈P (i) where Z is a problem specific normalization constant, P is the set of positive examples and (i) is the error term computed on i ∈ P.
Computation of the Loss. Given logits (s i ), L can be computed in three steps [6,28] (Fig. 2 green arrows): Step 1. The difference transform between logits s i and s j is computed by x ij = s j − s i .
Step 2. Using x ij , errors originating from each pair of examples are calculated as primary terms (L ij ): where p(j|i) is a probability mass function (pmf) that distributes (i), the error computed on i ∈ P, over j ∈ N where N is the set of negative examples. By definition, the ranking-based error (i), and thus L ij , requires pairwisebinary-ranking relation between outputs i and j, which is determined by the non-differentiable unit step function Using H(x ij ), different ranking-based functions can be introduced to define (i) and p(j|i): e.g. the rank of the ith example, rank(i) = j∈P∪N H(x ij ); the rank of the ith example among positives, rank and number of false positives with logits larger than s i , N FP (i) = j∈N H(x ij ). As an example, for AP Loss [6], using these definitions, (i) and p(j|i) can be simply defined as NFP(i) rank(i) and H(xij ) NFP(i) respectively [28].
Step 3. Finally, L is calculated as the normalized sum of the primary terms [28]: Optimization of the Loss. Here, the aim is to find updates ∂L ∂si , and then proceed with backpropagation through model parameters. Among the three computation steps ( Fig. 2 orange arrows), Step 1 and Step 3 are differentiable, whereas a primary term L ij is not a differentiable function of difference transforms. Denoting this update in x ij by ∆x ij and using the chain rule, ∂L ∂si can be expressed as: Chen et al. [6] incorporate the error-driven update [35] and replace ∆x ij by −(L * ij − L ij ) where L * ij is the target primary term indicating the desired error for pair (i, j). Both AP Loss [6] and aLRP Loss [28] are optimized this way.

Our Reformulation: Identity Update
We first identify two drawbacks of the formulation in Section 3.1: (D1) Resulting loss value (L) does not consider the target L * ij , and thus, is not easily interpretable when L * ij = 0 (cf. aLRP Loss [28] and our RS Loss -Section 4); (D2) Eq. 2 assigns a non-zero primary term only if i ∈ P and j ∈ N , effectively ignoring intra-class errors. These errors become especially important with continuous labels: The larger the label of i ∈ P, the larger should s i be.
Definition of the Loss. We redefine the loss function as: where * (i) is the desired error term on i ∈ P. Our loss definition has two benefits: (i) L directly measures the difference between the target and the desired errors, yielding an interpretable loss value to address (D1), and (ii) we do not constrain L to be defined only on positives and replace "i ∈ P" with "i ∈ P ∪ N ". Although we do not use "i ∈ P ∪ N " to model RS Loss, it makes the definition of L complete in the sense that, if necessary to obtain L, individual errors ( (i)) can be computed on each output, and hence, L can be approximated more precisely or a larger set of ranking-based loss functions can be represented.
Computation of the Loss. In order to compute L (Eq. 4), we only replace Eq. 2 with: in three-step algorithm (Section 3.1, Fig. 2 green arrows) and allow all pairs to have a non-zero error, addressing (D2).
Optimization of the Loss. Since the error of a pair, L ij , is minimized when (i) = * (i), Eq. 5 has a target of L * ij = 0 regardless of L. Thus, ∆x ij in Eq. 3 is simply the primary term itself: ∆x ij = −(L * ij − L ij ) = −(0 − L ij ) = L ij , concluding the derivation of our Identity Update.

Rank & Sort Loss
In order to supervise the classifier of visual detectors by considering the localisation qualities of the predictions (e.g. IoU), RS Loss decomposes the problem into two tasks: (i) Ranking task, which aims to rank each positive higher than all negatives, and (ii) sorting task, which aims to sort the logits s i in descending order wrt. continuous labels y i (e.g. IoUs). We define RS Loss and compute its gradients using our Identity Update (Section 3.2 - Fig. 2).
Definition. Given logits s i and their continuous labels y i ∈ [0, 1] (e.g. IoU), we define RS Loss as the average of the differences between the current ( RS (i)) and target ( * RS (i)) RS errors over positives (i.e. y i > 0): where RS (i) is a summation of the current ranking error and current sorting error: For i ∈ P, while the "current ranking error" is simply the precision error, the "current sorting error" penalizes the positives with logits larger than s i by the average of their inverted labels, 1 − y j . Note that when i ∈ P is ranked above all j ∈ N , N FP (i) = 0 and target ranking error, * R (i), is 0. For target sorting error, we average over the inverted labels of j ∈ P with larger logits (H(x ij )) and labels (y j ≥ y i ) than i ∈ P corresponding to the desired sorted order, * where [P] is the Iverson Bracket (i.e. 1 if predicate P is True; else 0), and similar to previous work [6], Computation. We follow the three-step algorithm (Section 3, Fig. 2) and define primary terms, L ij , using Eq. 5, which allows us to express the errors among positives as: where ranking (p R (j|i)) and sorting pmfs (p S (j|i)) uniformly distribute ranking and sorting errors on i respectively over examples causing error (i.e. for ranking, j ∈ N with s j > s i ; for sorting, j ∈ P with s j > s i but y j < y i ): Optimization. To obtain ∂LRS ∂si , we simply replace ∆x ij (Eq. 3) by the primary terms of RS Loss, L ij (Eq. 9), following Identity Update (Section 3.2). The resulting ∂LRS ∂si for i ∈ N then becomes (see Appendix A for derivations): Owing to the additional sorting error (Eq. 7, 8), ∂LRS ∂si for i ∈ P includes update signals for both promotion and demotion to sort the positives accordingly: Note that the directions of the first and second part of Eq. 12 are different. To place i ∈ P in the desired ranking, * We provide more insight for RS Loss on an example in Appendix A.

Using RS Loss to Train Visual Detectors
This section develops an overall loss function to train detectors with RS Loss, in which only the learning rate needs tuning. As commonly performed in the literature [17,18], Section 5.2 analyses different design choices on ATSS [44], a SOTA one-stage object detector (i.e. k = 1 in Eq. 1); and Section 5.3 extends our design to other architectures.

Analysis and Tuning-Free Design Choices
ATSS [44] with its classification, box regression and centerness heads is originally trained by minimizing: where L cls is Focal Loss [22]; L box is GIoU Loss [33]; L ctr is Cross-entropy Loss with continuous labels to supervise centerness prediction; and λ box = 2 and λ ctr = 1. We first remove the centerness head and replace L cls by our RS Loss head, in addition to the standard ones, is common in recent methods (e.g. centerness head for ATSS [44], IoU head for IoU-Net [15], and mask IoU head for Mask-scoring R-CNN [14]) to regress localisation quality and prioritize examples during inference (e.g. by multiplying classification scores by the predicted localisation quality). Sampling heuristics are also common to ensure balanced training. Such architectures use many hyper-parameters and are sensitive for tuning. (b) Training detectors with our RS Loss removes (i) aux. heads by directly supervising the classification (Cls.) head with continuous IoUs (in red & bold), (ii) sampling heuristics owing to its robustness against class imbalance. We use losses with similar range with our RS Loss in other branches (i.e. GIoU Loss, Dice Loss) also by weighting each by using classification scores, obtained applying sigmoid to logits. (c) Instead of tuning λ k t s, we simply balance tasks by considering loss values. With this design, we train several detectors only by tuning the learning rate and improve their performance consistently.
(Section 4), L RS , using IoU(b i , b i ) between a prediction box (b i ) and ground truth box (b i ) as the continuous labels: where λ box , the task-level balancing coefficient, is generally set to a constant scalar by grid search. Inspired by recent work [5,28], we investigate two tuning-free heuristics to determine λ box every iteration: (i) value-based: λ box = L RS /L box , and (ii) magnitude-based: where |·| is L1 norm,b and s are box regression and classification head outputs respectively. In our analysis on ATSS trained with RS Loss, we observed that value-based task balancing performs similar to tuning λ box ( ∼ 0 AP difference on average). Also, we use scorebased weighting [18] by multiplying the GIoU Loss of each prediction using its classification score (details are in Appendix B). Note that value-based task balancing and scorebased instance weighting are both hyper-parameter-free and easily applicable to all networks. With these design choices, Eq. 14 has only 1 hyper-parameter (i.e. δ RS in H(·), set to 0.50, to smooth the unit-step function) Fig. 3 presents a comparative overview on how we adopt RS Loss to train different architectures: When we use RS Loss to train the classifier ( Fig. 3(b)), we remove aux. heads (e.g. IoU head in IoU-Net [15]) and sampling heuristics (e.g. OHEM in YOLACT [1], random sampling in Faster R-CNN [32]). We adopt score-based weighting in box regression and mask prediction heads, and prefer Dice Loss, instead of the common Cross-entropy Loss, to train mask prediction head for instance segmentation due to (i) its bounded range (between 0 and 1), and (ii) holistic evaluation of the predictions, both similar to GIoU Loss. Finally, we set λ k t (Eq. 1) to scalar L k cls /L k t (i.e. L cls = L RS ) every iteration ( Fig. 3(c)) with the single exception of RPN where we multiply the losses of RPN by 0.20 following aLRP Loss.

Experiments
To present the contribution of RS Loss in terms of performance and tuning simplicity, we conduct experiments on seven visual detectors with a diverse set of architectures: four object detectors (i.e. Faster R-CNN [32], Cascade R-CNN [2], ATSS [44] and PAA [17] -Section 6.1) and three instance segmentation methods (i.e. Mask R-CNN [12], YOLACT [1] and SOLOv2 [40] -Section 6.2). Finally, Section 6.3 presents ablation analysis.  Table 1) and does not need a sampler, an aux. head or tuning of λ k t s (Eq. 1).

One-stage Object Detectors
We train ATSS [44] and PAA [17] including a centerness head and an IoU head respectively in their architectures. We adopt the anchor configuration of Oksuz et al. [28] for all ranking-based losses (different anchor configurations do not affect performance of standard ATSS [44]) and set learning rate to 0.008. While training PAA, we keep the scoring function, splitting positives and negatives, for a fair comparison among different loss functions.
Comparison with AP and aLRP Losses, rankingbased baselines: We simply replaced Focal Loss by AP Loss to train networks, and as for aLRP Loss, similar to our RS Loss, we tuned its learning rate as 0.005 due to its tuning simplicity. Both for ATSS and PAA, RS Loss provides significant gains over ranking-based alternatives, which were trained for 100 epochs using SSD-like augmentation [25] in previous work [6,28]: 1.8/2.2 AP gain for ATSS and 3.7/3.3 gain for PAA for AP/aLRP Loss (Table 2).
Comparison with Focal Loss, default loss function: RS Loss provides around ∼ 1 AP gain when both networks are equally trained without an aux. head (Table 2) and 0.6 AP gain compared to the default networks with aux. heads.
Comparison with QFL, score-based loss function using continuous IoUs as labels: To apply QFL [18] to PAA, we remove the aux. IoU head (as we did with ATSS), test two possible options ((i) default PAA setting with λ box = 1.3 and IoU-based weighting, (ii) default QFL setting: λ box = 2.0 and score-based weighting) and report the best result for QFL. While the results of QFL and RS Loss are similar for ATSS, there is 0.8 AP gap in favor of our RS Loss, which can be due to the different positivenegative assignment method of PAA (Table 2).

Comparison with SOTA
Here, we use our RS-R-CNN since it yields the largest improvement over its baseline. We train RS-R-CNN for 36 epochs using multiscale training by randomly resizing the shorter size within [480, 960] on ResNet-101 with DCNv2 [45]. Table 3 reports the results on COCO test-dev: Our RS-R-CNN reaches 47.8 AP and outperforms similarly trained Faster R-CNN and Dynamic R-CNN by ∼ 3 and ∼ 1 AP respectively. Although we do not increase the number of parameters for Faster R-CNN, RS R-CNN outperforms all multi-stage detectors including TridentNet [19], which has more parameters. Our RS-R-CNN+ (Section 6.1.1) reaches 48.2 AP, and RS-Mask R-CNN+ (Section 6.2) reaches 49.0 AP, outperforming all one-and multi-stage counterparts.

Multi-stage Instance Segmentation Methods
We train Mask R-CNN [12] on COCO and LVIS datasets by keeping all design choices of Faster R-CNN the same.
LVIS: Replacing the Cross-entropy to train Mask R-CNN with repeat factor sampling (RFS) by our RS Loss improves the performance by 3.5 mask AP on the long-tailed LVIS dataset (21.7 to 25.2 with ∼ 7AP r improvement on rare classes) and outperforms recent counterparts (Table 5). Table 2. RS Loss has the least number of hyper-parameters (H#) and outperforms (i) rank-based alternatives significantly, (ii) the default setting with an aux. head (underlined) by 0.6 AP, (iii) score-based alternative, QFL, especially on PAA. We test unified losses (i.e. a loss considering localisation quality while training classification head) only without aux. head. All models use ResNet-50. adopt cosine annealing with an initial learning rate of 0.006. Then, we remove (i) OHEM, (ii) semantic segmentation head, (iii) carefully tuned task weights (i.e. λ box = 1.5, λ mask = 6.125) and (iv) size-based normalization (i.e. normalization of mask head loss of each instance by the ground-truth area). Removing each heuristic ensues a slight to significant performance drop (at least requires retuning of λ t - Table 6). After these simplifications, our RS-YOLACT improves baseline by 1.5 mask AP and 3.3 box AP.
SOLOv2: Following Wang et al. [40], we train anchorfree SOLOv2 with RS Loss for 36 epochs using multiscale training on its two different settings: (i) SOLOv2-light is the real-time setting with ResNet-34 and images with size 448 × 448 at inference. We use 32 images/batch and learning rate 0.012 for training. (ii) SOLOv2 is the SOTA setting with ResNet-101 and images with size 1333 × 800 at inference. We use 16 images/batch and learning rate 0.006 for training. Since SOLOv2 does not have a box regression head, we use Dice coefficient as the continuous labels of RS Loss (see Appendix C for an analysis of using different localisation qualities as labels). Again, RS Loss performs better than the baseline (i.e. Focal Loss and Dice Loss) only by tuning the learning rate (Table 7).

Comparison with SOTA
We use our RS-Mask R-CNN (i.e. standard Mask R-CNN with RS Loss) to compare with SOTA methods. In order to fit in 16GB memory of our V100 GPUs and keep all settings unchanged, we limit the number of maximum proposals in the mask head by 200, which can simply be omitted for GPUs with larger memory. Following our counterparts [40,41]

Ablation Experiments
Contribution of the components: Replacing Focal Loss by RS Loss improves the performance significantly (1 AP - Table 9). Score-based weighting has a minor contribution and value-based task balancing simplifies tuning.
Robustness to imbalance: Without tuning, RS Loss can train models with very different imbalance levels successfully (Table 10): Our RS Loss (i) yields 38.5 AP on COCO with the standard random samplers (i.e. data is relatively balanced especially for RPN), (ii) utilizes more data when the samplers are removed, resulting in ∼ 1 AP gain (38.5 to 39.6 AP), and (iii) outperforms all counterparts on the long-tailed LVIS dataset (c.f. Table 5), where the imbalance is extreme for R-CNN (pos:neg ratio is 1 : 10470 - Table 10). Appendix C presents detailed discussion.
Contribution of the sorting error: To see the contribution of our additional sorting error, during training, we track Spearman's ranking correlation coefficient (ρ) between IoUs and classification scores, as an indicator of the sorting quality, with and without our additional sorting error (see Eq. 6-8). As hypothesized, using sorting error improves sorting quality, ρ, averaged over all/last 100 iterations, from 0.38/0.42 to 0.42/0.47 for RS-R-CNN.
Effect on Efficiency: On average, one training iteration of RS Loss takes around 1.5× longer than score-based losses. See Appendix C for more discussion on the effect of RS Loss on training and inference time.

Conclusion
In this paper, we proposed RS Loss as a ranking-based loss function to train object detectors and instance segmentation methods. Unlike existing ranking-based losses, which aim to rank positives above negatives, our RS Loss also sorts positives wrt. their localisation qualities, which is consistent with NMS and the performance measure, AP. With RS Loss, we employed a simple, loss-value-based, tuning-free heuristic to balance all heads in the visual detectors. As a result, we showed on seven diverse visual detectors that RS Loss both consistently improves performance and significantly simplifies the training pipeline.

A. Details of RS Loss
In this section, we present the derivations of gradients and obtain the loss value and gradients of RS Loss on an example in order to provide more insight.

A.1. Derivation of the Gradients
The gradients of a ranking-based loss function can be determined as follows. Eq. 3 in the paper states that Our identity update reformulation suggests replacing ∆x ij by L ij which yields: We split both summations into two based on the labels of the examples, and express ∂L ∂si using four terms: Then simply by using the primary terms of RS Loss, defined in Eq. 9 in the paper as: With the primary term definitions, we obtain the gradients of RS Loss using Eq. A.17. Gradients for i ∈ N . For i ∈ N , we can respectively express the four terms in Eq. A.17 as follows: concluding the derivation of the gradients if i ∈ N . Gradients for i ∈ P. We follow the same methodology for i ∈ P and express the same four terms as follows: • j∈N L ji = 0 (no error when j ∈ N and i ∈ P for L ji -see Eq. A.18), • j∈P L ij reduces to S (i) − * S (i) simply by rearranging the terms and j∈P p S (j|i) = 1 since p S (j|i) is a pmf: Combining these four cases together, we have the following gradient for i ∈ P: Finally, for clarity, we rearrange the terms also by using * concluding the derivation of the gradients when i ∈ P.

A.2. More Insight on RS Loss Computation and Gradients on an Example
In Fig. A.4, we illustrate the input and the computation of RS Loss. We emphasize that our Identity Update provides interpretable loss values when the target value is nonzero (Fig. A.4(b)). Previous work [6,28] fail to provide interpretable loss values.
Computation of the Loss. To compute the loss following the three-step algorithm in the paper, the first and the third steps are trivial (Fig. 2 in the paper), thus, here we present in Fig. A.5 (a) how the primary terms (L ij ) are computed for our example (Fig. A.4) following Eq. A. 18.
Optimization of the Loss.

B. Analyses
Section B.1 presents our experiments to validate our design choices and Section B.2 discusses the drawbacks of aLRP Loss, and how we fix them.

B.1. Analysis to Determine Design Choices in Localisation Loss
In this section, we provide our analysis on ATSS [44] to determine our design choices for localisation. First, as a baseline, we train ATSS network with the following loss function: where L RS is our Rank & Sort Loss, λ box is the task-level balancing coefficient and L box is the box regression loss. First, we investigate two tuning-free heuristics to determine λ box every iteration: (i) value-based: λ box = L RS /L box , and (ii) magnitude-based: where |·| is L1 norm,b and s are box regression and classification head outputs respectively. Table A.11 presents that value-based task balancing performs similar to tuning λ box (∼ 0 AP on average).
Secondly, we delve into L box , which is defined as the weighted average of the individual losses of examples: . To compute primary terms, Lij, one needs current errors, target errors and pmfs for both ranking and sorting, which are included in the figure respectively. In order to compute the target errors on a positive i ∈ P, the examples are first thresholded from si and the ones with larger (i.e. sj ≥ si) logits are obtained. Then, target rankings are identified using continuous labels. The ranking and sorting errors computed for the target ranking determines target errors, * R (i) and * S (i). The ranking and sorting losses, R(i) − * R (i) and S (i) − * S (i) respectively, are then distributed over examples causing these losses uniformly via pmfs pS(j|i) and pR(j|i) to determine pairwise errors, i.e. primary terms. (b) The gradients are obtained simply by using primary terms as the update in Eq. 3 following identity update yielding Eq. 11 and A.30 for negatives and positives respectively. Thanks to the novel sorting objective, RS Loss can assign a gradient to suppress a positive example when it is not ranked among positives accordingly wrt its continuous label (e.g. i = 1).
where L GIoU (b i , b i ) is the GIoU Loss [33], and w i is the instance-level importance weight. Unlike no prioritization (i.e. w i = 1 for i ∈ P), recently, a diverse set of heuristics assigns different importances over i ∈ P: centernessbased importance [38,44] aims to focus on the proposals (i.e. point or anchor) closer to the center of b i , score-based heuristic [18] uses the maximum of confidence scores of a prediction as w i , IoU-based approach [17] increases the losses of the predictions that are already better localized by w i = IoU(b i , b i ), and finally ranking-based weighting [28] uses where H(·) can be smoothed by an additional hyper-parameter (δ loc ). Note that these instance-level weighting methods perform similarly (largest gap is 0.2 AP - Table A.11) and we prefer score-based weighting with RS Loss.

B.2. A Comparative Analysis with aLRP Loss
In this section, we list our observations on aLRP Loss [28] based on our comparative analysis with RS Loss: Observation 1: Tasks competing with each other within the bounded range of aLRP Loss degrades performance especially when the models are trained 12 epochs following the common training schedule.
To illustrate this, we train Faster R-CNN [32] with aLRP Loss and our RS Loss using two different settings: • "Standard Training", which refers to the common training (e.g. [32,38,44]): The network is trained by a batch size of 16 images with resolution 1333 × 800 without any augmentation except the standard horizontal flipping. We use 4 GPUs, so each GPU has 4 images during training. We tune the learning rate of aLRP Loss as 0.009 and for our RS Loss we set it to 0.012. Consistent with the training image size, the test image size is 1333 × 800.
• "Heavy Training", which refers to the standard training design of aLRP Loss (and also AP Loss): The network is trained by a batch size of 32 images with resolution 512 × 512 on 4 GPUs (i.e. 8 images/GPU) using SSDsyle augmentation [25] for 100 epochs. We use the initial learning rate of 0.012 for aLRP Loss as validated in the original paper, and for our RS Loss, we sim-ply use linear scheduling hypothesis and set it to 0.024 without further validation. Here, following aLRP Loss (and AP Loss), the test image size is 833 × 500. 1. For both "heavy training" and "standard training", aLRP Loss has significantly lower performance after the first epoch (17.7 AP vs. 9.2 AP for heavy training and 22.0 AP vs. 14.4 AP) compared to RS Loss: aLRP Loss has a bounded range between 0 and 1, which is dominated by the classification head especially in the beginning of the training, and hence, the box regression head is barely trained. To tackle that, Oksuz et al. [28] dynamically promotes the loss of box regression head using a self-balance weight, initialized to 50 and updated based on loss values at the end of every epoch. However, we observed that this range pressure has an adverse effect on the performance especially at the beginning of the training, which could not be fully addressed by self-balance since in the first epoch the SB weight is not updated.
2. While the gap between RS Loss and aLRP Loss is 0.5 AP for "heavy training", it is 2.2 AP for "standard training". After the SB weight of aLRP Loss is updated, the gap can be reduced when the models are trained for longer epochs. However, the final gap is still large (∼ 2 AP) for "standard training" with 12 epochs since unlike aLRP Loss, our RS Loss (i) does not have a single bounded range for which multiple tasks compete, and (ii) uses an iteration-based selfbalance instead of epoch-based.
Observation 2: The target of aLRP Loss does not have an intuitive interpretation.
Self-balance (or range pressure -see Observation 1) is not the single reason why RS Loss performs better than aLRP Loss in both scheduling methods in Table A.12. aLRP Loss uses the following target error for a positive example i: * and τ is the positive-negative assignment threshold. However, unlike the target of RS Loss for specifying the error at the target ranking where positives are sorted wrt their IoUs (see Fig. A.5), the target of aLRP Loss does not have an intuitive interpretation. Observation 3: Setting τ in Eq. A.34 to the value of the positive-negative (anchor IoU) assignment threshold creates ambiguity (e.g. anchor-free detectors do not have such a threshold).
We identify three obvious reasons: (i) Anchor-free methods do not use IoU to assign positives and negatives, (ii) recent SOTA anchor-based methods, such as ATSS [44] and PAA [17], do not have a sharp threshold to assign positives and negatives, but instead they use adaptive thresholds to determine positives and negatives during training, and furthermore (iii) anchor-based detectors split anchors as positives and negatives; however, the loss is computed on the predictions which may have less IoU with ground truth than 0.50. Note that our RS Loss directly uses IoUs as the continuous labels without further modifying or thresholding them.
Observation 4: Using an additional hyper-parameter (δ loc ) for ranking-based weighting yields better performance for the common 12 epoch training.
As also discussed in Section B.1, ranking-based importance weighting of the instances corresponds to: aLRP Loss, by default, prefers not to smooth the nominator (H(x ki )) but rank(k) is computed by the smoothed unitstep function. We label this setting as "default" and introduce an additional hyper-parameter δ loc to further analyse ranking-based weighting. Note that the larger δ loc is, the less effect the logits will have on w i (Eq. A.32). In Table A.13, we compare these different settings on RS-ATSS trained for 12 epochs with our RS Loss, and observe that the default ranking-based weighting can be improved with different δ loc values. However, for our RS Loss, we adopt score-based weighting owing to its tuning-free nature.

C. More Experiments on RS Loss
This section presents the experiments that are omitted from the paper due to space constraints.
C.1. Effect of δ RS , the Single Hyper-parameter, for RS Loss. Table A.14 presents the effect of δ RS on RS Loss using ATSS. We observe similar performance between δ RS = 0.40 and δ RS = 0.75. Also note that considering positiveto-positive errors in the sorting error, we set δ RS different  Table  1 of the Paper A two-stage detector that uses random sampling and does not employ a method to adaptively set λ k t has at least 7 hyper-parameters since (i) for random sampling, one needs to tune number of foreground examples and number of background examples to be sampled in both stages (4 hyperparameters), and (ii) at least 3 λ k t s need to be tuned as the task-balancing coefficients in a loss with four components (Eq. 1 in the paper). As a result, except aLRP Loss and our RS Loss, all methods have at least 7 hyper-parameters. When the box regression losses of RPN and R-CNN are L1 Loss, GIoU Loss or AutoLoss, and the network has not an additional auxiliary head, 7 hyper-parameters aree sufficient (i.e. GIoU Loss [33], Carafe FPN [39] and AutoLoss-A [24]). Below, we list the methods with more than 7 hyperparameters:

C.3. Hyper-parameters of R-CNN Variants in
• FPN [21]   In order to provide more insight regarding the employment of continuous labels for the instance segmentation methods, we train YOLACT under four different settings: (i) without using continuous labels (c.f. "Binary" in Table  A.16) (ii) using IoU, the bounding box quality, as the continuous label (iii) using Dice coefficient, the segmentation quality, as the continuous label and (iv) using the average of IoU and Dice coefficient as the continuous label. Table  A. 16 suggests that all of these localisation qualities improve performance against ignoring them during training. Therefore, we use IoU as the continuous ground truth labels in all of our experiments with the exception of RS-SOLOv2, in which we used Dice coefficient, yielding similar performance to using IoU (Table A.16), since SOLOv2 does not have a box regression head.

C.5. Details of the Ablation Analysis on Different
Degrees of Imbalance successfully, and when more data is available (i.e. sampler is "None"), the performance improves from 38.5 to 39.6.
• When we train Mask R-CNN using RS Loss on the long-tailed LVIS dataset without any samplers, we observed that unlike COCO dataset, R-CNN training is extremely imbalanced (actual pos:neg task ratio is 1:10470) due to the large number of classes in LVIS dataset. Still, our RS Loss achieves SOTA performance despite this extreme imbalance (see also Table  5 in the paper).
As a result, we conclude that RS Loss can easily be incorporated to train data with different levels of imbalance.
Are Score-based Loss Functions Robust to Imbalance Without Tuning? Here, we investigate how Cross-entropy Loss and Focal Loss behave when samplers are removed without any tuning.
Cross-entropy Loss. As a fair baseline for our RS Loss, we use Faster R-CNN with GIoU Loss and only remove random sampling gradually similar to how we did for RS Loss. Table A. 18 shows that, as opposed to our RS Loss, the performance significantly drops once the samplers are removed for Cross-entropy Loss, and hence Cross-entropy Loss cannot be directly employed to train different levels of imbalance unlike our RS Loss.
Focal Loss. There are many design choices that one needs to tune in order replace the standard Cross-entropy Loss by Focal Loss. Here, instead of tuning each of these extensively, we use a commonly used setting in one-stage detectors [18,22,44] to train RPN and R-CNN. In particular, we use individual class-wise binary sigmoid classifier (as we also did for RS Loss), set the learning rate to 0.01, the weight of GIoU Loss to 2 and the bias terms in the last layer of the classification head 3 such that the confidence scores of the positives are 0.01 to prevent destabilization of the training due to large loss value originating from negatives. However, we observed that Focal Loss is not able to perform as good as Cross-entropy Loss and RS Loss with this configuration (Table A. 17). Hence, as a generalisation of Cross-entropy Loss, Focal Loss at least needs to be tuned carefully in order to yield better performance.
As a result, we conclude that common score-based loss functions (i.e. Cross-entropy Loss and Focal Loss) cannot handle different degrees of imbalance without tuning; while our RS Loss can.

C.6. Effect of RS Loss on Efficiency
We discuss the effect on efficiency on two levels: (i) training and (ii) inference. As a result, the models trained by our RS Loss outputs larger confidence scores, and accordingly, the score threshold needs to be adjusted accordingly for better efficiency. Table A.21. The performances of RS-Mask R-CNN and baseline Mask R-CNN (i.e. trained by Cross-entropy Loss) over different confidence score thresholds on LVIS v1.0 val set. RS-Mask R-CNN is robust to score threshold while the performance of Mask R-CNN degrades rapidly especially for rare classes. Our best method achieves 25.2 mask AP at 11.7 fps (bold), which is also slightly faster than the best performing method of Mask R-CNN (underlined). Accordingly, we set NMS score threshold of RS-Mask R-CNN to 0.60 for LVIS dataset. Inference time is reported on a single A100 GPU.