{'fraction_non_alphanumeric': 0.03575634804729825, 'fraction_numerical': 0.01889103500259104, 'mean_word_length': 4.881961762261014, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
MKIoU Loss: Towards Accurate Oriented Object Detection in Aerial Images

Oriented bounding box regression is crucial for oriented object detection. However, regression-based methods often suffer from boundary problems and the inconsistency between loss and evaluation metrics. In this paper, a modulated Kalman IoU loss of approximate SkewIoU is proposed, named MKIoU. To avoid boundary problems, we convert the oriented bounding box to Gaussian distribution, then use the Kalman filter to approximate the intersection area. However, there exists significant difference between the calculated and actual intersection areas. Thus, we propose a modulation factor to adjust the sensitivity of angle deviation and width-height offset to loss variation, making the loss more consistent with the evaluation metric. Furthermore, the Gaussian modeling method avoids the boundary problem but causes the angle confusion of square objects simultaneously. Thus, the Gaussian Angle Loss (GA Loss) is presented to solve this problem by adding a corrected loss for square targets. The proposed GA Loss can be easily extended to other Gaussian-based methods. Experiments on three publicly available aerial image datasets, DOTA, UCAS-AOD, and HRSC2016, show the effectiveness of the proposed method.


Introduction
As an emerging field, oriented object detection has recently seen rapid development. Different from general horizontal object detection, oriented object detection aims to locate arbitrary-oriented objects more accurately from complex backgrounds. It has been widely used in urban planning, surveying and mapping, and resource discovery. Although considerable progress has been made recently [1][2][3][4][5], challenges still exist, such as large aspect ratios, dense distribution, and arbitrary direction.
Most of the current detection algorithms are regression-based methods. Therefore, the design of the regression loss directly determines the performance of the detection. In generic horizontal detection, the common regression loss can be divided into two main types. One is Ln-norm-based, such as L1 loss, L2 loss, and Smooth L1 loss [6]. However, these methods optimize each parameter individually, ignore the correlation between the parameters, and are non-scale-invariant. Another is Intersection over Union (IoU)-based, such as GIoU [7], DIoU [8], CIoU [8], and EIoU [9]. The IoU-based losses are scaleinvariant, and the IoU calculation process implicitly encodes the association between the parameters. Furthermore, as the primary evaluation metric in object detection, mean Average Precision (mAP) depends heavily on the IoU scores between the predicted box and ground truth box. Therefore, the IoUbased losses have a natural advantage over the Ln-norm losses. For horizontal object detection, the IoU can be easily calculated by the coordinates of the bounding box. However, for oriented object detection, due to the diversity of the two oriented boxes, the precise SkewIoU calculation process is too complex to be computed directly under existing deep learning frameworks [10]. Moreover, the boundary *Corresponding author problems caused by the periodicity of the angle and the confusion of edges always affect the performance of the detection. Therefore, the design of loss function for oriented object detection is meaningful and gradually being emphasized. Some current methods [11][12][13] model the oriented bounding box into Gaussian distributions (see Eq.1) and then use the distance measure in statistics (such as Wassertein Distance, Kullback-Leibler Divergence, and Bhattacharyya Distance) to calculate the similarity, effectively avoiding the boundary problems. However, there is no clear correspondence between the calculated similarity and SkewIoU.
On this basis, Yang et al used Kalman filter to mimic the computing mechanism of SkewIoU, but the inconsistency between loss and evaluation metric still exists.
In this paper, we explore a more accurate modulated Kalman IoU of approximate SkewIoU for oriented object detection, named MKIoU. Through the analysis, we reveal that the inaccurate regression is due to the insensitivity of the loss variation tendency when the predicted box is close to the ground truth box, i.e., the inconsistency between the loss and the metric. Thus, fully considering the sensitivity of angle deviation and width-height offset to the loss variation, a modulation method is proposed to make the loss more consistent with the evaluation metric. Furthermore, the Gaussian modeling methods cannot learn the angle information of square targets, as shown in Fig 1 (top), which affects the performance of high-precision detection of the model. Thus, a GA Loss is proposed, which can correct the angle confusion of square objects, as shown in Fig 1 (down). Moreover, it can be easily added to other Gaussian modeling methods. The highlights of this paper can be summarized as follows: 1) A more accurate modulated Kalman IoU loss is proposed, named MKIoU, which is more concerned with the consistency of the loss trend with the evaluation metric. Compared to KFIoU [14], our method achieves higher accuracy.
2) A Gaussian Angle loss is presented to overcome the angle confusion problem of Gaussian modeling methods for square objects, thus further improving the performance of high-precision detection.
3) The proposed methods can be easily implemented in existing deep learning frameworks.
Extensive experimental results on three public datasets show the effectiveness of our proposed methods, which achieve comparable performance to the state-of-the-art methods on the DOTA datasets.

Related Work
In this section, we briefly review existing works on oriented object detection.

Oriented Object Detector
Most of the existing oriented object detectors are based on mature horizontal object detectors such as Fast R-CNN [6], Faster R-CNN [15], YOLO [16,17], SSD [18], and RetinaNet [19]. Compared to general object detection, oriented object detection is mainly applied to aerial images and scene text.
Arbitrary orientations, cluttered arrangements, and small objects are the main challenges for oriented object detection [2]. Some early methods [1,20,21] used multiple-angle anchors to detect oriented objects. However, the increase of anchors leads to a considerable increase in the calculation. RoI Transformer [3]proposed a rotated RoI (Region of Interest, RoI) learner, which can transform horizontal RoIs to rotated RoIs, avoiding a large number of rotated anchors. Gliding Vertex [4] proposed a new representation for oriented objects to avoid representation confusion. R 3 Det [5] designs a feature refinement module by pixel-wise feature interpolation to obtain more accurate features. CSL [22] and DCL [23] regard angular prediction as a classification task. ReDet [24] proposes a Rotation-equivariant detector to extract rotation-equivariant features, which can accurately predict the direction. Oriented RepPoints [25]employs a set of adaptive points to capture the geometric, spatial, and semantic scenario.
AOPG [26] first produces coarse-oriented boxes in an anchor-free manner and then refines them into high-quality-oriented proposals.

Regression Losses
The regression loss function is designed to continuously optimize the position of the predicted box to approach the ground truth box, thus directly determining the performance of the detector. For general object detection, regression loss has achieved relatively mature development, such as the commonly used Ln-norms losses (L1, L2, and SL1 [6]) and IoU losses (GIoU [7], DIoU [8], CIoU [8], and EIoU [9]). However, in oriented object detection, the complex and variable nature of the oriented box makes the boundary problem prominent. Recently, some improved methods for loss function have been proposed.
For Smooth L1 loss, to avoid learning confusion caused by the sequence of the points, RSDet [27] proposes a modulated rotation loss, and SBD [28] proposes a sequential-free box discretization method. RIL [29] transforms the bounding box regression problem into an optimal matching problem, solving the boundary problem caused by ambiguous representations. SCRDet [2] adds the IoU constant factor to Smooth L1 loss to solve the boundary problem. On this basis, Area-IoU [30] introduces an area-guided function and IoU-guided function to solve the problems of scale imbalance and angle periodicity problems. For approximate SkewIoU loss, PIoU [31] approximates the intersection area using the number of pixels in the overlapping area. To tackle the uncertainty of convex caused by rotation, RIoU [32] defines a projection operation to estimate the intersection area.
CFA [33] proposes a convex hull feature adaptation method based on CIoU loss for point-based detectors. In order to avoid the boundary problem of the oriented bounding box, GWD [11] first proposes to convert the oriented bounding box into a 2-D Gaussian distribution and then measures the loss by the Wasserstein distance between the two distributions, which effectively solves the boundary problem of the oriented box. However, when the object is close to a square, the angle information cannot be accurately learned. KLD [12], ProbIoU [13], and KFIoU [14] are also based on Gaussian modeling. However, the square problem still exists.

Method
In this section, we will describe the details of the proposed methods. We first analyze the inadequacy of the KFIoU Loss. Then a modulated Kalman IoU loss is proposed based on KFIoU, which is more concerned with the sensitivity of angular deviation and width-height offset to loss variation. Furthermore, Gaussian Angle loss is presented to overcome the angle confusion problem of square targets caused by Gaussian modeling methods.

Modulated Kalman Intersection over Union Loss
Firstly, to avoid the boundary problem, we follow the GWD [11] to convert the oriented bounding box ( , , , , ) x y w h  into 2-D Gaussian distributions ( , ) N   as follows: ( ) where R represents the rotation matrix, and  represents the diagonal matrix of eigenvalues. Then we can easily calculate the area of the oriented bounding box by its covariance: But the key is how to calculate the intersection area. Inspired by KFIoU [14], we use Smooth L1 loss to make the center points of two Gaussian distributions coincide, then use the Kalman filter to obtain the covariance pt  of overlapping regions of two Gaussian distributions: where p  is the covariance matrix of the predicted box, t  is the covariance matrix of the ground truth box. Combined with Eq. 2, we can obtain the approximate area pt S of the intersection of the two oriented bounding boxes. Then, the KFIoU can be expressed as: The KFIoU Loss is easy to implement in current deep learning frameworks. However, the result of KFIoU loss in the higher precision index (AP75 and AP50: 95) is not good, i.e., the regression results are not accurate. By combining Eq. 1 to Eq. 4, we have: It can be obtained that 4 AB +, the equation holds when one of the following conditions is satisfied: That is, when the predicted box overlaps completely with the ground-truth box, the KFIoU takes the maximum values of 1/3. We do a visualization analysis of KFIoU, as shown in Fig 2. Note that for convenience comparison, we multiply KFIoU by 3 so that the range of values is [0,1]. It can be found that there is a significant difference between the variation trend of KFIoU (red) and the actual trend of SkewIoU(blue). Specifically, when the predicted box is close to the ground truth box, KFIoU is much larger than the actual IoU, which makes the optimization of the model insensitive, thus damaging the high-precision detection performance of the model. We expect that the variation trend of the approximate IoU to be consistent with SkewIoU. Therefore, we propose MKIoU, which is adjusted by a modulation factor α: The value range of MKIoU is (0,1]. KFIoU is a special case of MKIoU, when α=1, MKIoU is equal to 3  KFIoU. Fig 2(a) shows the sensitivity of the width-height offset of MKIoU for different values of α when the two bounding boxes have the same center and angle. It can be seen that when α=1, the sensitivity of MKIoU to width-height offset is much lower than that of SkewIoU, which leads to inaccurate results. As α increases, the sensitivity increases gradually, making the trend of MKIoU more consistent with the evaluation metric. Fig 2(b) shows the sensitivity of the angle deviation of MKIoU when the two bounding boxes have the same center, width, and height. Similar conclusions can be drawn to the above. Note that with the increase of the aspect ratio of the object, the sensitivity of the SkewIoU to the angle increases, and a small angle change will also cause a sharp change in SkewIoU, which is not conducive to a stable and accurate regression for the loss function. Therefore, α is not the bigger the better, the final value of α is determined by comparative experiments.

Gaussian Angle Loss for angle confusion problem of Gaussian modeling
The Gaussian-based approaches effectively avoid the boundary problem, but when the object is a square, i.e., wh = , its 2D Gaussian distribution is a circle that does not accurately represent the orientation. Specifically, when wh = , then The angle information is missing. This will lead to the angle confusion of square-like objects, which will affect the performance of high-precision detection. Fig. 1 (top) shows the visualization of angle confusion problem. To solve this problem, we propose the Gaussian Angle loss, which can be expressed as:

Advanced detectors with the proposed loss
Our final regression loss can be expressed as: Our methods can be easily applied to the current advanced detectors on MMrotate [37], such as Rotated RetinaNet, R 3 Det [5], and RoI Transformer [3]. In this paper, we configure our loss on all three detectors and do corresponding experiments. The multi-task loss is: Focal loss [19] as classification loss cls L , and for RoI Transformer, we use Cross Entropy Loss.

Experiments
In this section, experiments are constructed to evaluate the performance of our MKIoU loss.

Datasets
Our experiments evaluate on DOTA1.0 [34], UCAS_AOD [35], and HRSC2016 [36] datasets. UCAS_AOD is a car and airplane dataset that contains 1,510 images with 14,596 instances. We randomly select 1,057 images for training and 453 images for testing.
HRSC2016 is a ship dataset that contains 1061 images with 2976 instances. The image size ranges from 300 × 300 to 1,500 × 900 pixels. We randomly select 617 images for training and 444 images for testing. All images were resized to 800 × 800 for training and testing.

Implement details
We use MMrotate [37] to conduct all our experiments where many advanced rotation detectors are integrated. We use Pytorch [38] with 1 GeForce RTX 2080 Ti for single-scale training and 1 NVIDIA A100 for multi-scale training. Experiments are initialized by ResNet50 [39] by default unless otherwise specified. We use SGD optimizer to train the model. The momentum and weight decay are 0.9 and 0.0001, respectively. The batchsize is 2 (2 images per GPU). The initial learning rate is 2.5e-3 by default. For DOTA datasets, we trained 12 epochs in total, and the learning rate is reduced by a factor of 10 at 8 epochs and 11 epochs, respectively, and doubled if data augmentation or multi-scale training is used. For UCAS_AOD, we trained 12 epochs in total, and the learning rate is reduced by a factor of 10 at 8 epochs and 11 epochs, respectively. For HRSC2016, we trained 72 epochs in total, and the learning rate is 1.25e-3 and reduced by a factor of 10 at 48 epochs and 66 epochs.

Evaluation metrics
mAP is the most critical metric for evaluating the performance in object detection. It can be divided into COCO [40]

Ablation study 1) hyperparameter experiments
We perform detailed hyperparameter experiments on the proposed loss in this paper. Table 1 provides the results of the experiments under different α and β on RetinaNet. Note that KF Loss means 1-KFIoU, f (KFIoU) refers to the nonlinear transformation. Follow the paper [14], ( 2) Ablation study of the proposed methods on different datasets Table 2 shows the ablation experiments of the proposed method on UCAS_AOD. We performed experiments on two oriented detectors, RetinaNet and R 3 Det. It can be seen from the table that the proposed MKIoU is better than KFLoss and f (KFIoU) in terms of high precision, which indicates that adjusting width-height sensitivity and angle sensitivity is beneficial to more accurate regression.
Furthermore, the proposed GA loss further improved the accuracy. Combined with Fig 2, we can conclude that the GA loss can effectively correct the angle confusion problem of square objects caused by Gaussian modeling. It should be noted that the number of planes accounts for half in UCAS_AOD and its bounding box is close to square. Therefore, the high-precision results of Gaussian-based methods on this dataset are poor.   Table 3 shows the high-precision experimental comparison of our method with other Gaussianbased losses on HRSC2016. GWD and KLD are currently advanced loss functions in high-precision detection, which is also based on Gaussian modeling. From the table, we can see that KFIoU can achieve comparable results with GWD, but there is still a big gap with KLD. We analyze that this is due to the targets in HRSC2016 are ships with a large aspect ratio, while KFIoU is insensitive to angle changes. To make the loss more sensitive to angle changes, we increase the α, but being too sensitive will lead to unstable model training, so we finally set α=2.5 in HRSC2016. Our method greatly improves the detection accuracy (On RetinaNet, 11.84%, 7.89%, 5.73% improvement on AP75, AP85, and mAP, respectively) and achieves comparable results to KLD. Similar results can be concluded in To better demonstrate the effectiveness of the proposed methods, we conducted ablation experiments on the DOTAv1.0 dataset with two different detectors, as shown in Table 4. The result of (KFIoU) f is better than KFLoss, but the high precision index is still not well. Our proposed MKIoU maintains comparable performance on AP50 and shows a significant improvement on AP75 and mAP, with an increase of 1.34% and 0.73% on RetinaNet, respectively. After adding GA Loss, AP75 and mAP are further improved by 2.37% and 1.00%, respectively. Even on the better detectors (RoI Transformer), our method still improves the high precision detection performance of the model.  The generality of Gaussian Angle Loss. Table 5 shows the comparison results of other Gaussian-based losses (KLD and GWD) after adding GA loss under different detectors on UCAS_AOD. It can be seen from the table that the proposed Gaussian angle loss is equally effective for KLD and GWD and dramatically improves the detection accuracy, which suggests that the GA Loss is generalizable for Gaussian-based loss.

3) Comparison with the State-of-the-Arts Methods
Results on HRSC2016. Table 6 shows the comparison of the proposed method with the other advanced methods on HRSC2016 datasets. The proposed MKIoU achieves the best results 90.45% (VOC 2007 metric) on ResNet 101, which indicates that our method is effective. Results on DOTA. To further demonstrate the robustness of the proposed method, we extensively compare the proposed method with other advanced methods on the more challenging DOTA dataset.
As shown in Table 7, our method achieves 77.76% mAP in single-scale strategy and 80.83% mAP in multi-scale strategy, which is comparable to the current state-of-the-arts methods. Compared with KFIoU, the AP50 of our method is 0.6% higher. The visualization of detection results is shown in Fig 5. Furthermore, Table 8 shows the high-precision results on DOTAv1.0. In single-scale strategy, our method is slightly higher than KFIoU on AP50 but significantly better on AP75 and AP50:95, with an increase of 1.71% and 0.78%, respectively. This fully shows that our method has more advantages in high-precision detection. In multi-scale strategy, our method achieves 80.83%, 55.46%, and 51.46% mAP on AP50, AP75, and AP50:95, respectively. Table 6. Comparison of the mAP (VOC 07 metric) of different methods on HRSC2016

Conclusion
In this paper, we proposed a modulated Kalman IoU Loss of approximate SkewIoU for oriented object detection, named MKIoU Loss. Compared with KFIoU, MKIoU pays more attention to the sensitivity of width-height offset and angle deviation to loss variation, thus significantly improving the high-precision detection performance of the model. Furthermore, Gaussian Angle loss is presented to overcome the angle confusion problem for square objects of the Gaussian modeling method.
Experimental on several public datasets demonstrate the effectiveness of our approach and achieve comparable results to state-of-the-art methods. Although our method achieves more accurate detection, there are still shortcomings, such as misclassification and detection of objects with huge aspect ratios.
Therefore, in future work, we will focus on effectively reducing misclassification and improving objects' precision with huge aspect ratios.