{'fraction_non_alphanumeric': 0.03640281679301189, 'fraction_numerical': 0.012236887674606534, 'mean_word_length': 4.875650011304544, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Points as Queries: Weakly Semi-supervised Object Detection by Points

We propose a novel point annotated setting for the weakly semi-supervised object detection task, in which the dataset comprises small fully annotated images and large weakly annotated images by points. It achieves a balance between tremendous annotation burden and detection performance. Based on this setting, we analyze existing detectors and find that these detectors have difficulty in fully exploiting the power of the annotated points. To solve this, we introduce a new detector, Point DETR, which extends DETR by adding a point encoder. Extensive experiments conducted on MS-COCO dataset in various data settings show the effectiveness of our method. In particular, when using 20% fully labeled data from COCO, our detector achieves a promising performance, 33.3 AP, which outperforms a strong baseline (FCOS) by 2.0 AP, and we demonstrate the point annotations bring over 10 points in various AR metrics.


Introduction
Object detection is one of the fundamental problems in computer vision. Modern object detectors [12,14,15,22,29] have achieved great success with the help of tremendous annotated data. However, it is very costly to annotate a large amount of detection data. Specifically, for each object instance, a precise bounding box needs to be labeled manually and carefully, which is quite time-consuming: it takes 10-35 seconds [27,24,1] for labeling an object.
To reduce the cost of data annotation, weakly supervised object detection (WSOD) and semi-supervised object detection (SSOD) methods are proposed. Weakly supervised object detection methods [2,11,25,36] utilize large data with weak annotations, such as image labels, which is far easier to collect than precisely annotated bounding boxes. The semi-supervised object detection methods [10,17,26,28,31] learn detectors with a small amount of box-level labeled images and large unlabeled images, * Equally contribution. † Corresponding author.
where the cost of image annotation is small. Although these methods can reduce the cost of annotation significantly, their performance is far inferior to their supervised counterparts [14,15,29]. To make a trade-off between annotation cost and performance, weakly semi-supervised object detection methods (WSSOD) [33] are studied, which use small box-level labeled images as well as large weakly labeled images to learn detectors. However, image-level annotations in weakly annotated data are not optimal for object detection task since image labels do not contain the instance-level information of all objects. Motivated by [1], we annotate each instance in the image by one point (as shown in Figure 1d) instead of image-level annotation, for two main reasons. Firstly, compared with image-level annotation, points bring much richer information, not only the category of the object but also the strong prior of object location. Secondly, there is no strict requirement on point annotations, such as center points of objects. Thus, the increase in the cost of labeling is marginal compared with the image-level annotation [1]: 23.3 sec/image vs. 20.0 sec/image in VOC dataset [6].
Though the above new setting is better for weakly semisupervised object detection, most recent detectors [14,15,29] have difficulty in predicting object boxes based on point annotations. In most detectors, FPN [14] is a basic component, which utilizes multi-level feature maps to predict object boxes. FPN can boost the performance of detectors, but it is incompetent to predict object boxes using point annotations since it is difficult to select the optimal box prediction from multi-level ones, predicted for a point annotation. For the single-level feature detectors, they may suffer from bad performance [20,21,22] or strict requirement on point annotations [5,12,35]   Besides, it inherits the strong representation of DETR, having a good performance on object detection. But, different from DETR, we encode position and category of annotated points into object queries with the point encoder, which easily establishes one-to-one correspondences between points and object queries, being fit for box prediction based on points. In addition, to boost detection performance and make optimization easier, we do box predictions as offsets w.r.t. point position rather than make box predictions directly like DETR.
To show the superiority of our detector, we mainly evaluate our proposed detector on the MS-COCO dataset [16]. To make a fair comparison, we take FCOS [29] as the default baseline, which is regarded as a point-based detector. Following our proposed weakly semi-supervised object detection setting, object instances of small image data fraction (5% ∼ 50%) are annotated fully and the rest are annotated by points. In these various settings with a different fraction of fully-annotated image data, our proposed detector outperforms other modern detectors, including multi-level feature detectors and single-level feature detectors. In particular, when using 20% fully labeled data from COCO, our detector outperforms FCOS and Faster R-CNN by 2.0 AP and 1.9 AP, respectively.
Our main contributions can be summarized as follows: • We propose a potential and novel setting for the weakly semi-supervised object detection task, which comprises small fully annotated images and large weakly annotated images by points. Compared with the image-level data setting [1], this setting introduces weakly instance-level information with marginal annotation cost, which is fit for object detection. This provides a new perspective to improve detection performance with weakly annotated detection images.
• Based on the above setting, we analyze the drawbacks of existing modern object detectors and propose Point DETR, which is simple and easily implemented. The proposed detector takes object points as input, transforms these points into object queries, and predicts object box precisely for these queries, as shown in Figure 3.
• Extensive experiments on COCO dataset [16] are conducted to demonstrate the effectiveness of our proposed detector. Our detector outperforms most modern detectors in various data settings. We also do quantity and quality experiments to show our detector solves the problems suffered by most modern detectors.

Related Work
Supervised Object Detection: With the large-scale fully annotated detection data, existing modern detectors [12,14,15,22,29] have obtained great improvements in the object detection task. These detectors can be divided into two categories: two-stage detectors and one-stage detectors. FPN [14] is a popular two-stage detector, which predicts object proposals firstly and refines these proposals finally. Unlike two-stage detectors, one-stage detectors [12,15,29] directly outputs the classification and location of each object without refinement. Though achieving great success, these detectors are trained with a large amount of fullyannotated data, which is costly to annotate. Thus, there are many works proposed to reduce the annotation cost.
Semi-Supervised/Weakly Supervised Object Detection: Semi-supervised object detection (SSOD) [10,17,26,28,31] and weakly-supervised object detection (WSOD) [2,11,25,36] are introduced to reduce the large cost of data annotation. The semi-supervised object detection methods learn detectors with a small amount of box-level labeled images and large unlabeled images. Jeong et al. [10] employ consistency constraints for object detection to exploit unlabeled data. While, weakly supervised object detection methods utilize large data with weak annotations, such as image labels. Bilen et al. [2] learn an object detector under image-level supervision by combining region classification and selection. Furthermore, pursuing the performance of supervised detection and keeping the low cost of annotation, weakly semi-supervised object detection methods (WSSOD) [33] are studied, which use small box-level labeled images as well as large weakly labeled images to learn detectors. Unlike these semi-/weakly-supervised object detection, our proposed detector utilizes a new low-cost annotation: points, which provide instance location. Recently, UFO 2 [23] also uses point supervision as weak labels, but it does not explore the point information sufficiently as we shown in Section 4.3.
Point based Semi-Supervised Segmentation: Point supervision [1,19,34] has been employed by semantic segmentation. Bearman et al. [1] incorporate point supervision along with objectness prior to boost segmentation performance and alleviate annotation burden. Qian et al. [19] leverage semantic relationships among several labeled points to address the semantic scene parsing task. Different from these works, we focus on object detection task, where point-based detection has been explored little. Due to a lack of exploitation, existing detectors do not fit pointlevel annotation well.
DETR: Unlike existing detectors, DETR [4] removes the need for many hand-designed components like a nonmaximum suppression procedure or anchor generation. By virtue of Transformer [30], DETR takes an image as input and directly outputs a fixed set of box predictions. For the point-based detection task, DETR has a beneficial characteristic: a single-level feature map, avoiding the multi-level selection problem. However, directly applied DETR into point-based detection task is not practical. Object queries in DETR are general embeddings and have no specific point information. Conversely, our detector encodes the position and category of annotated points into object queries with the point encoder and establishes one-to-one correspondences between point annotations and object queries.

Method
In this section, we first introduce the task of weakly semi-supervised object detection (WSSOD) with point annotations and discuss why existing object detectors can not fit this task well. Next, in order to solve it, we illustrate our novel detector, Point DETR, in detail.
WSSOD with point annotations: WSSOD generally uses a small set of instance-level labeled images and tremendous weakly image-level labeled images as training data (Figure 1c). However, for object detection, image-level labeled images do not fit WSSOD well, since it can not provide instance information. This raises a natural question: is there a new data annotation for weakly labeled images, which has instance information without a large annotation burden? In this paper, we introduce point annotation for weakly labeled images.
Point Annotations: It is introduced by Bearman et al. [1] for weakly semantic segmentation, but it has not been explored well in object detection. In object detection, we define point annotation as follows: it locates on the object and takes object class as its category. Thus, we represent an object as (x, y, c), where (x, y) ∈ [0, 1] 2 and c represent point location and object category, respectively. We must note that our method is robust to point location, as shown in Table 1e. Therefore, the point annotations can locate at the anywhere of objects. In this way, we can alleviate the annotation burden.
Overall Framework: With this new setting that a small number of supervised images and a large number of weakly supervised images, we adapt self-training as our default training pipeline, which has made considerable progress in semi-supervised learning (e.g. Lee [13],Noise-Student [32],STAC [26]). The steps are summarized as follows: 2. Generate pseudo-labels of weakly point annotated images using the trained teacher model. 3. Train a student model with fully labeled images and pseudo-labeled images.
The overall framework is shown in Figure 2. For most self-training based detection methods, hyper-parameters are selected carefully since they must keep true object boxes and screen out false ones as much as possible. Instead, we can directly predict the corresponding object box for each point annotation without duplicate object boxes. Although choosing hyper-parameters is no longer an obstacle to performance, predicting object boxes from point-level annotations with existing detectors remains a problem.
Discussion on Existing Detectors: Existing detectors can be divided into two categories: multi-level feature detectors and single-level feature detectors. For multi-level detectors (e.g. FCOS [29]), it is difficult for them to predict object boxes with point annotations since point annotations do not have feature-level information, which is used to select one prediction from multi-level box predictions (Figure 8b). On the other hand, single-level feature detectors (e.g. Faster R-CNN [22]) suffer from the bad performance or strict requirement on point annotations though avoiding choosing feature map levels (Figure 8c). For more experiments see Section 4.3.

Point DETR
To avoid the drawbacks of existing detectors in the WS-SOD with point annotations task, we introduce a novel detector, Point DETR: adding a point encoder to DETR. It transforms point annotations into object queries, extracts image features for each object query, and outputs the corresponding object box. Next, we introduce a key element of Point DETR, point encoder, which is critical to the WSSOD with point annotations task.

DETR:
We begin by reviewing DETR [4], which is an endto-end set-based object detector. DETR consists of a CNN backbone, an encoder-decoder transformer, and a prediction head. DETR first extracts a single-level 2D feature map from the CNN backbone, flattens it, and supplements it with a positional encoding. Then, the encoder-decoder transformer takes as input a fixed set of object queries (learned positional embeddings) and attends to 1D image feature embeddings. Finally, the output embeddings of the transformer are passed to the prediction head that predicts either a detection (class and bounding box) or a "no object" class.
Point DETR: Point DETR, as shown in Figure 3, adopts most components of DETR. To fit the point annotated images, Point DETR has a special module, point encoder. Point encoder can encode the point annotations into object queries, which are taken as input by the transformer decoder. Unlike the object queries in DETR that are learned positional embeddings, these object queries are specific instance embeddings which contain position and category information of object instances. Thus, these object queries have a one-to-one correspondence with object instances. Moreover, the number of object queries varies with the number of object instances in an image instead of a fixed number(e.g. 100) like DETR.
During training, we simply define the loss of each object query as L = L box , since we already have category for each object query and only need to regress the object box. The bounding-box loss L box is identical as it defined in DETR. But, for the box predictionb i , it calculated bŷ is the location of point annotation and ∆b i ∈ [0, 1] 4 is the relative offsets w.r.t. the point location (x, y) following FCOS [29]. In our experiments, we show this way of regression can alleviate the mismatch between point annotation and object box, see Section 4.3.
Point Encoder: In point DETR, how to encode point annotations into object queries is critical for point encoder. As shown in Figure 4, a point annotation (x, y, c) is decomposed to a 2D coordinate (x, y) ∈ [0, 1] 2 and category index c. Based on (x, y), the position embedding e pos ∈ R 256 is extracted from fixed spatial positional encodings [30,18,4], which is the same as one used in the transformer encoder. For category embedding e cat ∈ R 256 , it is obtained from predefined learnable category embeddings by category index, i.e. c. In the end, we fuse these embedding to get the object query by sum operation.
Though point encoder is simple and easily implemented, it bridges the divisions between point annotations and object queries. In the experiments, we show the essentials of every component (positional encoder and category encoder) in point encoder, see section 4.3.

Experiments
We evaluate our models on the COCO 2017 detection dataset [16] with synthetic point annotations (details in section 4.1). We report the standard COCO metrics including AP (averaged over IoU thresholds), AP 50 , AP 75 . In addition, to show the quality of generated pseudo-boxes, we also calculate the mIoU between the generated pseudo-boxes and the ground truth bounding boxes.
With existing detectors that can not be directly applied to our point annotated settings, we make some modifications to existing detectors: FCOS and Faster R-CNN. These modified detectors are denoted as FCOS † and Faster R-CNN † , respectively. For FCOS † , we separately extract point features from multi-level feature maps by bilinear interpolation [9] and predict the corresponding object box, finally use the box prediction with the highest point category score as the pseudo-box. As for Faster R-CNN † , we extract point features from one-level feature map, and then predict boxes  for different anchors, finally use one with the highest point category score as the pseudo-box.

Implementation Details
We use ResNet-50 [8] as the default backbone for different detectors and set the hyper-parameters following these detectors.
Dataset: We train the model with 118k training images and evaluate the performance of the detectors on the remaining 5k val images. Specially, for our point annotated setting, we randomly sample 5%, 10%, 20%, 30%, 40%, 50% of training images as the fully labeled set and use the rest of the images as a weakly labeled set. In this paper, we noted them as different data settings for simplicity, e.g. 20% data setting. For the weakly labeled set, we synthesize the point annotations for each object as follows: (a) if the object has instance segmentation, randomly sample a point from the instance mask as the point annotation for the object; (b) if not, simply randomly sample a point in its bounding box.  For the training of the teacher model, it is simple for FCOS † and Faster R-CNN † . We train them with their default training settings. For a fair comparison, we also use data augmentation as shown in [4]. For Point DETR, it follows most of the training settings used in [4] with several differences: we train the model for 108 epochs on 8 GTX 1080Ti GPUs, with 2 images per GPU. To ensure training stability, we use a warmup scheme [7] in the first epoch. The learning rate is reduced by a factor of 10 at epoch 72 and 96, respectively. In the training, we randomly sample a point in each bounding box and transform points into point annotations. With these point annotations, we train Point DETR as shown in Figure 3.
For the default student model, we combine the fully labeled images and pseudo-labeled images generated by the teacher model to train the student, as showed in Figure 2.

Main Results
We first show the effectiveness of Point DETR on different data split settings, see Figure 6. We train the student model (i.e. FCOS) only with the fully annotated images (noted as "Supervised"). By comparing "Supervised" with the student model trained with pseudo-boxes, we can evaluate the benefits brought by the pseudo-boxes. Point DETR and FCOS † outperform "Supervised" by a large margin. This demonstrates that images with point annotations can improve the performance of the detection task. Furthermore, Point DETR outperforms FCOS † by a considerable margin.
Next, we verify the factors that contribute to the great performance of our method. We compare the accuracies of FCOS and DETR as shown in Figure 7, DETR performs worse than FCOS in most settings. Given that our method based on DETR achieves greater performance, we can conclude that the high accuracy of our method does not mainly 20 Figure 7. Comparison in APs of FCOS and DETR to demonstrate the improvement comes from our method rather than a stronger teacher model. FCOS trained in DETR augmentation for a fair comparison. In most cases (5% ∼ 40%), FCOS has a better performance than DETR.
benefit from its strong representation. Moreover, we conduct quality and quantity experiments to show the superiority of our method on pseudo-object boxes, see Figure 8. FCOS † , a multi-level feature detector, can not predict object boxes well due to FPN, and Faster R-CNN † , a singlelevel feature detector, also has difficulty regressing box owing to poor representation. But, Point DETR can generate a more precise object box than other detectors. Specifically, the mIoU of Point DETR is larger than FCOS † and Faster R-CNN † by 6.3 and 5.3, respectively. Based on the above experiments, our method achieves considerable performance mainly by generating precise pseudo-object boxes from point annotations.

Ablation Experiments
We conduct the ablation experiments at 20% data setting. Results are shown in Table 1 and discussed in detail next.
Point Encoder: Table 1a shows the effectiveness of the components in the point encoder module (as we shown in Figure 4) . Point DETR with only positional embeddings outperforms one with only category embeddings and point DETR has a severe loss in AP (18.6 points) without positional embeddings. Based on that our method only regresses the object boxes, this suggests that it is difficult to learn the relative offsets of a point with respect to the bounding box without positional embeddings. We also find that adding category embeddings to positional embeddings can boost the performance by 2 points. We conjecture this improvement is caused by the that category embeddings can provide object prior, such as object shape.   we replace FCOS with RetinaNet [15]. In Table 1b, we find that our method has a 2.1 AP gain over baseline. This demonstrates that our method is robust to the student model.

Single-Level Detector:
We compare Point DETR with single-level feature detectors and choose Faster R-CNN † as the default single-level feature detector. As shown in Point Location: To validate that our method is robust to the point location, we compare performances between two point location schemes: center point and arbitrary point on objects. As shown in Table 1e, our method has comparable performance between these two point location scheme.
Absolute vs. Relative Regression: Our method use relative regression to predict object boxes. In Figure 5, we compare our relative regression with absolute regression used in DETR. Absolute regression incorrectly matches the point with the bounding box that does not correspond (e.g. the green clock in Figure 5b) in some cases. Compared with absolute regression, relative regression has little mismatch problem between point and object box, we attribute it to its use of the prior knowledge: the point is in the bounding box.
Point Annotations: To evaluate the effectiveness of point annotations, we compare our approach with the method without point annotations. For a fair comparison, we use DETR as the method without point annotations. We apply a self-training framework (following [26]) on DETR directly. We train DETR with only fully labeled images first, and then generate pseudo-boxes for weakly labeled images without point annotations. To remove duplicate boxes, we use a threshold τ = 0.7 which results in the best box predictions on weakly labeled images. For the generated pseudo-object boxes, they do not have the one-to-one correspondence with point annotations. Thus, it is impractical to calculate the mIoU between generated boxes and ground truth boxes. To make comparison available, we use standard COCO metrics instead of mIoU , as shown in Table 1f. Point DETR performs on par with DETR on mAP and outperforms DETR by a large margin in the recall. Specifically, Point DETR achieves over 10 points of improvements in various AR metrics (e.g. AR s , AR m , AR l , AR 100 ) and its AP is comparable with DETR (26.8 vs. 26.9). Though Point DETR is 3.3 points AP 75 lower than DETR, which is possibly explained by that high τ screens out low-quality boxes and remains high-quality boxes, the higher recall of Point DETR can offset this bad influence. Additionally, we set the classification score of pseudoboxes generated by DETR to a constant value like 0.5, which is consistent with our method. In this setting, the performance of DETR drops by a large margin and performs much worse than our method. This highlights that with point annotations, our method does not suffer from the quality of classification score.
We also analyze the errors of the generated boxes by TIDE [3] in Figure 9. Missed ground truths is the largest  . Diagnosing the errors of generated pseudo-boxes by TIDE [3]. Different error types: Cls: localized correctly but classified incorrectly, Loc: classified correctly but localized incorrectly, Both: both cls and loc error, Dupe: duplicate detection error, Bkg: detected background as foreground, Miss: missed ground truth error.
issue for DETR, while it does not affect the performance of Point DETR greatly. This is explained by that with point annotations, Point DETR does not miss objects like DETR. In addition, unlike DETR, location error is the main challenge of Point DETR. Also, Point DETR also has duplicate detection errors. This is caused by those point annotations residing in multiple bounding boxes that would predict object boxes for wrong ground truths, which results in a ground truth that has multiple box predictions.

Conclusion
In this work, we verify the effectiveness of point annotations in the weakly semi-supervised detection task. We also show that the power of point annotations is hindered by existing detectors. In order to solve this, we propose the Point DETR which applies a point encoder to the point annotations to establish the one-to-one correspondence between point annotations and objects. Our approach is simple and implemented easily. We demonstrate its efficacy by the extensive experimental analysis showing that it achieves state-of-the-art performance.