{'fraction_non_alphanumeric': 0.03215019995029484, 'fraction_numerical': 0.014075597026727818, 'mean_word_length': 4.911847201816482, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Open-Set Semi-Supervised Object Detection

Recent developments for Semi-Supervised Object Detection (SSOD) have shown the promise of leveraging unlabeled data to improve an object detector. However, thus far these methods have assumed that the unlabeled data does not contain out-of-distribution (OOD) classes, which is unrealistic with larger-scale unlabeled datasets. In this paper, we consider a more practical yet challenging problem, Open-Set Semi-Supervised Object Detection (OSSOD). We first find the existing SSOD method obtains a lower performance gain in open-set conditions, and this is caused by the semantic expansion, where the distracting OOD objects are mispredicted as in-distribution pseudo-labels for the semi-supervised training. To address this problem, we consider online and offline OOD detection modules, which are integrated with SSOD methods. With the extensive studies, we found that leveraging an offline OOD detector based on a self-supervised vision transformer performs favorably against online OOD detectors due to its robustness to the interference of pseudo-labeling. In the experiment, our proposed framework effectively addresses the semantic expansion issue and shows consistent improvements on many OSSOD benchmarks, including large-scale COCO-OpenImages. We also verify the effectiveness of our framework under different OSSOD conditions, including varying numbers of in-distribution classes, different degrees of supervision, and different combinations of unlabeled sets.


Introduction
The success of deep neural networks relies on large collections of labeled data, although creating such large-scale datasets is expensive and time-consuming. The recent development of successful Semi-Supervised Learning (SSL) methods alleviates this requirement by making better use of unlabeled data, narrowing the performance gap between the SSL models and the fully-supervised model.
Inspired by SSL methods for image classification [1,23,39,45,54,52,11,14,40], recent works on semi-supervised object detection (SSOD) [41,29,55,44] have applied the self-training method, which trains the model with the pseudo-labels of the unlabeled data. These works often consider a scenario where the labeled set is randomly sampled from a dataset (e.g., MS-COCO [27]) and use the remaining images as the unlabeled set. This implicitly assumes the label spaces of labeled and unlabeled data are identical. However, this closed-set assumption is unlikely  [40,29] rely on thresholding for filtering unreliable pseudo-labels, some OOD objects in unlabeled images are mispredicted as inlier objects with high confidence. These OOD objects are then mis-labeled, trained on, and lead to worse performance.
to happen in real-world situations, where unlabeled images collected in the wild might contain out-of-distribution (OOD) objects, which are unseen, undefined, and unknown in the available labeled set.
We are thus interested in a more practical yet challenging problem, Open-Set Semi-Supervised Object Detection (OSSOD), which aims to leverage the unconstrained unlabeled images (i.e., images containing unseen OOD objects), to improve an object detector trained with the available labeled data as shown in Figure 1a. When adding unlabeled data containing open-set categories, we observe that the existing successful SSOD method leads to a lower performance gain or even degraded results. This is different from the common belief that SSL methods can benefit from using more unlabeled data. We attribute the above phenomena to the semantic expansion issue, where OOD objects are mispredicted as in-distribution objects with high confidence and misused as pseudo-labels with confidence thresholding (Figure 1b).
To eliminate the detrimental effect of OOD samples, we propose to add an additional OOD filtering process into the existing SSOD training pipeline. More concretely, we first consider online OOD detectors to perform OOD filtering. An online OOD detector is a prediction head/branch we straightforwardly add on the object detector using existing OOD methods [13,24,15,47]. However, we find that such methods cannot produce satisfactory results due to interference with other tasks, e.g., bounding box localization and box classification (See Section 4.2 for further discussion). In order to address this, we propose a simple but effective strategy that uses an offline OOD detection module, which is disentangled from the architecture of the object detector. This OOD detector is based on a self-supervised DINO [2] model, and it provides several advantages. First, the pre-training of DINO does not require label annotations, so it is suitable for the low-label setting and alleviates the concern of limited amounts of labels for OOD detection tasks. Secondly, it is more effective in detecting OOD objects in the pseudo-labels compared with other (online) OOD methods as shown in Section 5.2, and this eliminates the detrimental effect of OOD samples for OSSOD tasks. Lastly, since the architecture of OOD detector and object detector are independent, the training of the two models can be done separately and prevent interference as we observed in online OOD detectors.
In our experiments, we first provide a systematic analysis between several OOD detection methods, and our results suggest that offline methods show consistent improvements over the online detection methods. We also show that using the offline OOD detector can filter the OOD objects in pseudo-labels and consistently improve against the existing SSOD methods under different open-set scenarios, including under different combination of unlabeled sets, varying number of in-distribution (ID) classes, and different number of images. We also find that using the pseudo-labels generated from our framework is even more effective than using the ground-truth labels provided in OpenImages (see Section 5.3).
We highlight the contributions of this paper as follows: -To the best of our knowledge, we are the first to address the open-set semisupervised object detection tasks, and we analyze the limitation of existing SSOD methods on OSSOD tasks. -We identify the challenges in designing OOD detection in OSSOD tasks, present online and offline OOD detectors, and provide a systematic comparison between these two modules. -Through our extensive experiments, we demonstrate that an offline OOD detector can effectively remove OOD objects in pseudo-labels, and this leads to a significant improvements under different OSSOD scenarios, including a varying number of ID/OOD classes, different degrees of supervision, and different scales of datasets.

Related Work
Open-set semi-supervised object detection focuses on improving an in-distribution object detector with in-the-wild unlabeled data and preventing the detrimental effects caused by distracting OOD objects in a more practical semi-supervised setup. This task is different from the existing works on open-vocabulary/openworld object detection [53, 10,37,56,21,20,18], where the goal is to improve the recall and accuracy of novel/OOD objects. Semi-Supervised Object Detection. Recent successful methods on semisupervised learning for image classification have applied various data augmentation and consistency regularization on unlabeled data [1,23,39,45,54,52,11,14,40]. These techniques have also promoted the development of semi-supervised learning in object detection tasks. For example, several works [41,29,55,44] apply the selftraining method, which is based on the Teacher-Student framework. For example, STAC [41] uses the labeled data to train the Teacher, which generates pseudolabels to supervise the Student model. To refine the quality of pseudo-labels, existing works propose several techniques, including exponential moving average (EMA) [55,49,29,50], co-rectify mechanism [55], replacing hard pseudo-labels with soft pseudo-labels [44], and addressing class imbalance in pseudo-labels [29]. While promising results have been made, existing SSOD works usually experiment on datasets where the labeled and unlabeled sets have the same object categories. We are interested in a more challenging scenario, where some objects in the unlabeled set are novel and never appear in the labeled set.
Out-of-Distribution Detection. Out-of-distribution (OOD) classes are object categories that do not appear in the training label space and are not known a-priori, and OOD detection, which plays a vital role in OSSOD tasks, is a binary classification problem that decides whether a sample is an ID or OOD. Early works use different scoring functions to estimate the likelihood of a sample being OOD, including maximum softmax probability (MSP) [12,25] and Mahalanobis distance [24]. Another line of works [13,28,32] exploit large sets of OOD samples during training and they generally achieve better performance on detecting the outlier objects. There are also several generative-based methods [58, 34,36,33] for OOD detection tasks, though it is difficult and prohibitively challenging to apply them on large-scale semi-supervised object detection tasks. Among the existing OOD works, Fort et al. [8] shows state-of-the-art performance on OOD detection by simply fine-tuning the pretrained vision transformer with the available inlier data. While promising results have been made on OOD detection tasks, most of the existing works only experiment on small-scale image classification tasks, and their extension to the large-scale object detection tasks is not verified. The only work on OOD detection for object detection is VOS [7], but the interference with pseudo-labeling in SSOD tasks and computation cost of virtual negative sampling make it incompatible with the existing SSOD methods.

Revisiting Semi-Supervised Object Detection
Semi-supervised object detection (SSOD) aims to learn an object detector by using a set of labeled images [41,29,55,49] apply the self-training method and have shown significant improvements in this semi-supervised scenario. The typical strategy of such methods is to generate pseudo-labels (i.e., pseudoboxes and the corresponding class labels) of unlabeled images and then train the object detector using both labeled data and unlabeled data with pseudo-labels.
These works adopt the Teacher-Student framework, where the Teacher model generates pseudo-labels to train the Student model. Specifically, the Teacher model takes the weakly-augmented unlabeled data as input and generates the box predictions, and the bounding boxes with confidence higher than a pre-defined threshold τ are selected as the pseudo-boxes. The Student then takes as input the same images with stronger augmentation, and we enforce the consistency loss L unsup (i.e., unsupervised loss) between its predictions and the generated pseudo-labels. To train the Student, we combine both the supervised loss L sup and the unsupervised loss L unsup .
where λ is the unsupervised loss weight andŷ u i = δ(ỹ u i ; τ ) represents the pseudo-labels, which are derived from the bounding box predictionỹ u i after the confidence thresholding function δ(·) with the pre-defined threshold τ .
To verify the effectiveness of the models, existing works often experiment on a setting where the labeled and unlabeled sets are randomly sampled from the same dataset. This implicitly assumes the set of object categories are the same across the labeled and unlabeled sets, and we formalize this setup as the closed-set SSOD.

Open-Set Semi-Supervised Object Detection
While the above self-training methods show promising results in closed-set SSOD, it restricts the set of object categories in the unlabeled set to be the same as the labeled set. This setting, however, is less practical since unlabeled datasets collected in the wild might contain images with novel objects beyond what were presented in the labeled dataset. We are thus interested in OSSOD, a more practical yet challenging problem setup, where novel and undefined objects appear in the unconstrained unlabeled set and are not available in the labeled set with a limited amount of data. Fig. 2. Illustration of semantic expansion. (a) With the object detector trained with the labeled data, some OOD objects in unlabeled data are predicted as ID objects, and solely using the confidence thresholding based on the box score cannot effectively suppress these OOD objects in pseudo-labels. Using the noisy pseudo-labels in selftraining methods makes the open-set issue become more severe after several training iterations, and (b) thus the false-positive rate (i.e., percentage of OOD objects predicted as ID) increases over time. We first analyze the existing successful SSOD work 1 on OSSOD tasks. Interestingly, we found it shows limited performance gain in the open-set scenario compared with closed-set scenario as shown in Figure 3. We attribute this lower performance gain to semantic expansion as presented in Figure 2a. To be more specific, when OOD/novel objects exist in the unlabeled set, the closed-set classifier (e.g., ROIhead classifier in Faster-RCNN [35]) might mispredict these OOD instances as ID objects with high confidence (a similar observation is also made recently in prior works [5,31]). These over-confident and incorrect OOD instances are prone to be selected as the pseudo-labels of ID objects even after confidence thresholding, and using these incorrect OOD pseudo-labels in the training makes the EMA-updated Teacher mispredict more OOD objects as ID objects. After the iterative EMA updates between the Teacher and Student, the semantic discrimination of ID objects is enlarged and incorrectly covers more OOD objects. This causes the false-positive rate of the OOD pseudo-labels to increase as the model is trained longer, as presented in Figure 2b.

Semantic Expansion in Open-Set Scenarios
To address the above issue and improve the performance on OSSOD tasks, we propose an integrated framework to detect the OOD instances in the pseudo-boxes and eliminate their detrimental effect during semi-supervised learning. With the

Raw Predictions
Remove Low-confidence Boxes Combined with the existing closed-set SSOD methods, OOD detection aims to remove the OOD objects in pseudo-labels and prevent semantic expansion in OSSOD. For the online OOD detector, we add an additional branch on the object detector, but it suffers from the instability of pseudo-labels used for SSOD and leads to degraded performance of the object detector. In constrast, the offline OOD detector does not have these limitations and produce better results on OOD detection due to its nature of independent model architecture.

Offline OOD Detection
goal of deciding whether a sample is an ID or OOD data (i.e., binary classification), prior works have proposed several OOD detection methods [12,24,28,13,38], although most of these works only verify the effectiveness of the methods on the image-level vision tasks. To further advance to OOD detection for instance-level tasks (e.g., object detection), we consider online and offline OOD detectors in the following sections.

Online OOD detection
As illustrated in Figure 4, online OOD detection expands the architecture of the existing object detector so that it can perform OOD detection in addition to bounding box classification and localization. One simple way to detect the OOD objects is to use the maximum softmax probability (MSP) [12] of ROIhead classifier as an indicator for identifying OOD samples. In a similar spirit, another way is to apply a range of existing OOD detection methods [12,24,28,13,38,47] and add an additional OOD branch on the ROIhead. We combine the semi-supervised object detection loss L ssod as mentioned in Eq. 3 and OOD detection loss L ood defined in the original works, and we train the model in an end-to-end manner.
However, these online OOD detectors have several limitations in OSSOD tasks. On the one hand, as presented in Figure 5a, pseudo-labeling used for SSOD significantly degrades the performance of online OOD detection (we use MSP [12] as an example), and such a trend is hypothetically caused by the instability of pseudo-labels (i.e., classification noise in pseudo-labels). On the other hand, bundling two different tasks (e.g., OOD detection and object detection) in a shared architecture leads to sub-optimal performance of object detection task as shown in Figure 5b, making the online OOD detection less favorable to be applied for OSSOD tasks.

Offline OOD detection.
To ameliorate the above issues, as shown in Figure 4, we alternatively propose to use offline OOD detection, where we construct an OOD detector that is disentangled from the architecture of the object detector and not jointly trained with the object detector. This framework design provides two advantages: 1) the offline OOD detector is compatible with any existing SSOD methods [41,55,29,57], since the offline OOD detector is modularized and independent from the object detector. 2) Such a framework design alleviates the concern about competing task objectives between OOD detection and object detection.
To design the offline OOD detector, we exploit the current state-of-the-art OOD method, which is simple yet effective on image-level OOD tasks. Specifically, as pointed out by Fort et al. [8], one highly successful method for OOD detection is to simply fine-tune a pretrained ViT-B [6] with available ID data. Despite its promising results, pre-training a ViT-B requires ImageNet-21k images and the corresponding large amount of annotated labels, which is not suitable for our semi-supervised setting and would result in an unfair comparison to prior works. DINO as an offline OOD detector. We therefore utilize the self-supervised DINO [2], which is only trained with ImageNet-1k images without using any label annotations 2 . To perform the OOD detection, we fine-tune DINO with multi-class training and then compute different OOD scores (described below) to decide whether an object is an ID or an OOD sample. To be more concrete, the OOD detector is trained as a simple classifier with K + 1 output classes (K is the number of ID object classes), where the additional node is for the OOD and novel samples. Unlike prior works [32,13,28] that used a large amount of additional OOD data during training, we do not assume the availability of such data. We therefore only use the available ID labeled data and propose to regard the background proposal patches as the OOD samples. Specifically, we crop the image patches according to the ground-truth boxes, and the background patches are randomly sampled from the proposal boxes labeled with the low IoU to the ground-truth boxes. We compute the OOD detection loss L ood with the crossentropy which enforces the ID patches as the corresponding ID foreground labels and enforces the background patches as the OOD class. While it is sub-optimal to use background instances as OOD class, they still provide sufficient negative gradients for distinguishing the ID objects when the novel OOD objects are not available in the labeled set.
After the fine-tuning of the DINO model, we freeze the DINO model and experiment the following common scores for OOD detection: ( where f (x) represents the DINO intermediate feature vector of patch x,μ k is the estimated mean vector of class k, andΣ is the estimated covariance matrix. Bothμ k andΣ are estimated by using the available labeled data.
2. Inverse Abstaining Confidence [32,46]: wherep K+1 is the prediction confidence of K + 1-th class (i.e., abstention class) from the DINO classifier. 3. Energy Score [28]: where T is the temperature value, K is the number of classes, f i (x) represents the i-th index of the logit corresponding to the class i. We also consider Shannon Entropy and Euclidean Distance for the OOD scores. To decide whether an object is an ID or OOD class, we set a threshold δ ood on the OOD score, and the objects with OOD scores lower than the threshold are regarded as OOD samples.
Integration with Semi-Supervised Object Detection. With the goal of filtering the OOD objects in pseudo-labels, we integrate the trained OOD detector with the SSOD method by inserting the OOD filtering after the confidence thresholding as shown in Figure 4. In other words, the pseudo-boxes are derived by sequentially applying both confidence thresholding and OOD filtering on predicted boxes from Teacher model, and they are then used to compute the unsupervised loss defined in Eq. 3 to train the Student. Note that our OOD detector is complementary to the existing semi-supervised object detection works [41,55,29,57] and can also be combined with these SSOD methods to address the open-set issue.

Experimental Setting and Datasets
COCO-Open. COCO2017-train contains 117k images with box-level labels from 80 object categories. We randomly sample 20/40/60 classes as the ID classes and the remaining classes as the OOD classes. Specifically, since each image contains multiple objects, COCO2017-train is divided into the pure-ID, mixed, and pure-OOD image sets. Each image in the pure-ID set has at least one ID object and no OOD object, and each image in the pure-OOD set has at least one OOD object and no ID object. As for the mixed set, each image has at least one ID object and at least one OOD object. To make sure that OOD objects are not in the labeled set, we randomly sample the pure-ID images as a labeled set and the remaining data as the unlabeled set. Thus, the unlabeled set contains images from the pure-ID, pure-OOD, and mixed sets. COCO-OpenImages. To examine all methods in a large-scale OSSOD experiment, we use COCO2017-train as the labeled set and OpenImagesv5 [22] as the unlableed set. OpenImagesv5 contains 1.7M images with 601 object categories, and most object categories do not exist in COCO (and are hence OOD).
We also include more experiments in the supplementary material.

Comparison between online and offline OOD detection
We first present a comparison between online and offline detectors on OOD detection tasks (object detection results shown in Section 5.3), and we evaluate them with the standard OOD detection metrics, including area under the ROC curve (AUROC) and false-positive rate under the true-false rate with x% (FPR@TNRx). We consider several OOD detection methods [12,24,28,13,38,38,15,47] 3 . For a fair and thorough comparison, we adapt these methods from image classification to object detection tasks and implement them in a unified framework with the same implementation details such as batch size and optimization. To better understand the efficacy of each OOD method and prevent the interference of pseudo-labeling from affecting the performance, we do not use the pseudo-labels of unlabeled data and only use the available labeled data to train all OOD detectors (adding pseudo-labels worsen online OOD detection methods as we discussed in Section 4.2). As presented in Table 1, we list our observations as follows, 1) offline OOD detectors are significantly better than online OOD detectors on all evaluation metrics, and this trend also supports that the disentanglement between OOD detector and object detector can alleviate the interference between OOD detection and object detection. It is worth repeating that the feature backbone of Faster-RCNN is pretrained on ImageNet1k with label annotations, so the superiority of the DINO on OOD detection does not come from using extra images and labels. In addition, as mentioned in Section 4.3, another merit of the offline OOD detector is its robustness to the instability to pseudo-labeling used in SSOD due to the isolated training of OOD detectors and object detectors.
2) Among the OOD scores used in offline detectors, simply applying the Euclidean distance performs the bests on all metrics. The popular Mahalanobis distance cannot lead to satisfactory results due to the inaccurate covariance matrix estimated limited amounts of labeled data.
Additionally, in the supplementary material, we also list other in-depth analyses and discussions on OOD detection, including using supervised ViT-B as an offline OOD detector, and other offline OOD detection methods with different ID/OOD class splits and different degrees of supervision. Owing to the superior performance on OOD detection, we apply the offline OOD detectors in the following OSSOD experiments.

Experiments on Open-set Semi-Supervised Object Detection
In addition to the OOD detection performance shown in the previous section, we further advance to OSSOD tasks and assess the object detection performance under different OSSOD conditions, including different combinations of unlabeled sets, varying numbers of ID classes, different degrees of supervision, and largerscale OSSOD tasks (COCO-additional and COCO-GOI). Effect of different combinations of unlabeled sets. We experiment with the existing SSOD method (UT [29]) and our proposed OOD detectors by using different combinations of unlabeled data, including the pure-ID, the mixed, and the pure-OOD sets. The purpose of this experiment is to verify the existence of open-set issue/semantic expansion and investigate whether the OOD samples affect the SSOD methods. As shown in Figure 6, UT achieves 22.10 mAP when the pure-ID images are used in the unlabeled set, while the performance drops when the mixed and pure-OOD sets, containing OOD objects, are added. This shows that OOD objects in the unlabeled set do affect the effectiveness of the existing SS-OD method, and the trend is also contrary to the common belief that semi-supervised methods can derive more performance gains by using more unlabeled data. Therefore, with our proposed OOD filtering (OF-DINO), the performance of the object detector does not degrade but instead improves by using the mixed set, and this reflects the importance of eliminating the detrimental effect of OOD objects. Varying numbers of ID classes. We also experiment by varying the number of ID classes in COCO-Open, and we first find that UT shows less performance gain against label-only baseline when there are fewer ID classes (see Table 2). This is because the semantic expansion/open-set issue becomes more severe when more OOD objects appear in the unlabeled set, and such an issue can be alleviated by the OOD filtering. With the OOD filtering, we can obtain a larger improvement gain against UT, and it improves 6.54 mAP against the labeled-only baseline and 5.06 mAP against UT in the case of 20 ID classes. This verifies the efficacy of the OOD filtering using the different number of ID/OOD classes. Different degrees of supervision. In addition, we consider varying the number of labeled images in COCO-Open as shown in Table 3, and we find that using the OOD detector can also consistently improve against the label-only baseline and UT under different numbers of labeled images. When fewer labeled images are used, we could obtain more improvement gain by filtering OOD samples. Analysis on pseudo-labels. To better understand how our OOD detector helps the learning of the SSOD method in the open-set condition, we use the ground-truth labels to measure the number of ID/OOD boxes in pseudo-labels  (note that we only use ground-truth labels for the analysis and do not use them in the training). As illustrated in Figure 7, our OOD detector can effectively suppress OOD objects in pseudo-labels without sacrificing the ID objects, and thus this alleviates the semantic expansion issue mentioned in Section 4.1 and leads to improvement for the OSSOD task. COCO-GOI. We also consider a more challenging experiment, where we use unlabeled OpenImagesv5 to improve the object detector trained on COCO2017-train. Although OpenImagesv5 is substantially larger than COCO datasets, the object categories in OpenImagesv5 (601 classes) are more diverse than the object categories in MSCOCO (80 classes). This implies there are more distracting OOD objects that can affect the performance of existing SSOD methods. In addition to UT, we also consider a fully-supervised baseline, which uses ground-truth labels from both COCO2017-train and OpenImagesv5 to train an object detector. To be more specific, we first manually label the correspondence between 601 classes in OpenImagesv5 and 80 classes in COCO2017-train, and we only use labels of 80 COCO-classes in OpenImagesv5. We present the details of the correspondence in our supplementary material. As presented in Table 4, with our OOD filtering mechanism (DINO) we can improve UT from 41.81 mAP to 43.14 mAP. Another interesting result is that our framework can perform even better than the model with the ground-truth labels from OpenImages, and this indicates that using pseudo-labels generated from our framework might be more effective than the ground-truth labels provided in OpenImages. This is potentially caused by noisy labels from human annotations. In addition, we also find that some COCO object categories are rare in OpenImages, and this potentially limits further improvements by exploiting OpenImages as an unlabeled set (more discussion in the supplementary material).
Limitations and future works. While addressing OSSOD, we do not address other issues such as covariate shift and mismatch in object category distributions between datasets. The offline OOD detector is an individual module from the object detector, so it requires more computational resources in the training stage. However, this concern does not exist as we remove the offline OOD detector and only use the object detector in the inference stage. Our key message is that combining an offline OOD detection module and an SSOD method is a simple yet effective solution to address OSSOD tasks. Based on this integrated framework, there will be more advanced techniques for both SSOD and OOD detection methods, which can potentially improve the performance on OSSOD tasks.

Conclusion
Unlike traditional closed-set Semi-Supervised Object Detection, where the majority of the unlabeled images are from in-distribution datasets, we aim to train object detector with unconstrained unlabeled images -Open-Set Semi-Supervised Object Detection (OSSOD). In our proposed setting, unlabeled training images can contain object classes that are out-of-distribution, i.e., classes not included in the labeled set. To the best of our knowledge, this is the first work investigating OSSOD. We present comprehensive studies to understand the challenges for OSSOD and analyze why the performance of existing SoTA methods degrade even when trained with more unlabeled images. To overcome the challenges, we consider both online and offline OOD detectors and show that a simple yet effective offline OOD detector can further improve the SSOD models when training with large-scale unlabeled datasets, e.g., COCO-OpenImage. We achieved state-of-the-art results across existing and new benchmark settings, and some interesting directions such as more accurate and efficient OOD detectors are worth exploring for future research. A COCO-additional.
To examine OOD filtering in a large-scale scenario, we consider COCO-additional which aims to improve the fully-supervised object detector with the additional large-scale dataset (e.g., COCO2017-unlabeled). As presented in Table 7, UT (with data augmentation from SoftTeacher [49]) can achieve 44.06 mAP, and using the proposed OOD filtering can further improve UT and achieve 45.14 mAP and shows state-of-the-art result against the existing SS-OD works [44,55,19,43,49] on COCO-additional. This demonstrates that removing OOD objects from the large-scale unlabeled data can still improve the existing SSOD framework. However, it is worth noting that our proposed OOD filtering mechanism is not restricted to UT, and we believe that is also complementary to other SSOD methods [41,44,55,19,43,49]. To show the effectiveness of the OOD filtering, we show the pseudo-labels generated with and without OOD filtering in Figure 8. Without the OOD filtering, some OOD objects (e.g., fish) are predicted as inlier (e.g., bird) with high confidence. Such an issue is alleviated by the OOD filtering, which effectively suppresses the OOD objects in pseudo-labels and thus improves the accuracy of the object detector.

C Experiment results when using ViT-B as OOD detector
To understand how the ViT-B pretrained on ImageNet21k performs on OSSOD tasks, we examine the ViT-B model on the experiment setups presented in the main paper. Specifically, we consider different degrees of supervision in Table 5, different numbers of ID objects in Table 6, large-scale SSOD setting (e.g., COCOadditional) in Table 7, large-scale OSSOD setting (e.g., COCO-OpenImage) in Table 8. We observe using the ViT-B model can consistently lead to better results in all experiment scenarios, while, as we mentioned in the main paper, ViT-B requires large-scale supervised pre-training dataset (ImageNet21k), which is not suitable for our label-efficient settings. However, it does demonstrate that the method can scale with larger-scale pre-trained models, including when using in-the-wild unlabeled data (e.g. OpenImages). We first pre-select 20 COCO classes as ID classes and 60 COCO classes as OOD classes, and 1/2/4k images are randomly selected from the purely-ID set as the labeled set. The remaining images from pure-ID, pure-OOD, and mixed sets are used as the unlabeled set. We run each method 3 times and report the standard deviation.  Table 6. Mean average precision of COCO-Open when varying the number of ID objects. We first randomly sample 20/40/60 COCO classes as ID classes and remaining COCO classes as OOD classes, and 4k images are randomly selected from the purely-ID set as the labeled set. The remaining images from pure-ID, pure-OOD, and mixed sets are used as the unlabeled set. We run each method 3 times and report the standard deviation.

D Experiments on STAC
In addition to Unbiased Teacher [29] experimented with in the main paper, we also consider another SSOD method, STAC [41], to show that our findings are general. As shown in Table 9, we observe STAC also suffers from open-set issues  Table 9. Generalization of our findings to other methods, namely STAC [41] performance comparison between closed-set SSOD and open-set SSOD. For closed-set SSOD, we randomly select 1%/2% from the training set (i.e., 1172/2234 labeled images). For the open-set SSOD, we randomly samples 20 classes as ID classes and the remaining classes as OOD classes; hence the differences in performance of "Labeled only". We then sample the same amount of labeled images in both cases for a fair comparison. when experimenting on OSSOD tasks. Compared with the traditional closed-set SSOD task, the performance gain of STAC is smaller in open-set conditions.
To address the open-set issues, we also apply our proposed OOD detection method on STAC. As presented in Table 10, when the OOD detector (DINO) is applied, we can improve STAC from 18.60 mAP to 19.80 mAP. This shows that our proposed OOD filtering method is not restricted to any particular SSOD method and can potentially improve other SSOD methods.
Furthermore, similar to Unbiased Teacher [29], other existing SSOD methods [44,55,50,49] also applied confidence thresholding to select pseudo-labels, so they are also prone to suffer from the semantic expansion issue as we described in the main paper.

E Complete Comparison of OOD Detection
To compare OOD detection methods, we list a more complete comparison as presented in Table 11. We list other observations as follows: Mahalanobis Distance under limited amount of data setting With limited amount of training data (i.e., 4k ID images), Mahalanobis distance becomes Table 10. OOD filtering improves STAC on COCO-Open. We randomly sample 40 COCO classes as ID classes and remaining COCO classes as OOD classes, and 4k images are randomly selected from the purely-ID set as the labeled set. The remaining images from pure-ID, pure-OOD, and mixed sets are used as the unlabeled set. unreliable compared with other OOD metrics, and this is contrary to the prior works on OOD detection [8,24], where a large amount of data is used for training the OOD detectors. As computing Mahalanobis distance requires the covariance matrix and the class-wise mean vectors, and estimating the covariance matrix for high-dimension features is difficult and inaccurate, this is even more difficult when the data is scarce (especially when the number of instances is closer to or even less than the number of feature dimensions). This is also why we observe that Euclidean distance, which is equivalent to the Mahalanbois distance with the identity convariance matrix, leads to even better results than the Mahalanobis distance.
Another weakness of Mahalanobis distance is that it requires one to obtain class-wise mean and co-variance features by deriving the features for all ID images in the labeled set, so it is computationally slow (also pointed out in MOS [17]) and thus less preferable for large-scale open-set semi-supervised learning, which requires detecting OOD samples in each training iteration. Inverse abstaining confidence under different number of ID/OOD classes. Compared with other offline OOD detection metrics, using inverse abstaining confidence is more robust when varying the number of ID classes. To be more specific, as shown in Table 11, the Energy score and Shannon entropy perform on par or even better than the inverse abstaining confidence in the case of using 20 ID classes. However, as shown in Figure 9, when we increase the number of ID classes, the inverse abstaining confidence degrades much less than the Energy score and Shannon entropy. Such a property makes the inverse abstaining confidence more suitable for different OSSOD scenarios.

F Comparison between COCO-OpenImage and COCO-additional
In the main paper, we considered two large-scale unlabeled sets, OpenImagev5 and COCO2017-unlabeled, to improve the object detection trained on the labeled COCO2017-labeled. Our OOD filtering framework improves the supervised object detector from 40.90 mAP to 43.14 mAP by using OpenImagev5 as an unlabeled set and achieves 45.14 mAP when the COCO2017-unlabeled is used as an unlabeled set. As OpenImagev5 has more unlabeled images than COCO2017-unlabeled (1.7M vs. 120k), we are curious why the model using the OpenImagev5 as an unlabeled set cannot outperform the model using the COCO-unlabeled as an unlabeled set. Table 11. Evaluation of OOD detection for object detection tasks. We sample 20 classes from COCO as in-distribution (ID) objects, and 4000 pure-ID images are selected as labeled images. All methods are evaluated on COCO2017-val. Value before the slashes indicates ignoring background patches when computing AUROC and FPR, and value after the slashes regradining background patches as OOD objects when computing AUROC and FPR. We attribute this trend to the following factors: (i) Mismatch in class distribution. We first compare the class distribution of both datasets, and we find these two datasets have very different object distributions as shown in Figure 10a. The mismatch in the class distribution in the unlabeled set is prone to affect the frequency or confidence of objects predicted for the evaluation set, and this potentially leads to performance degradation in the evaluation set.
(ii) Some COCO objects are rare in OpenImage. When OpenImagev5 is used as an unlabeled set to train the object detector in a semi-supervised manner, we observe, as shown in Table 12, the performance on some objects are even lower than the supervised model due to the scarcity of these objects. Specifically, even though OpenImagev5 has more images than COCO2017-train, the number of some COCO objects in entire OpenImage are even fewer than the objects in COCO2017-train, as shown in Figure 10b. This suggests the objects are very rare and infrequently appear, and such a property potentially limits the further improvement by using OpenImagev5. Note that COCO2017-unlabeled follows the same class distribution as COCO2017-labeled (described in COCO official page), and both datasets have similar amount of images (120k vs. 117k). Table 12. Performance degradation of minor objects in semi-supervised learning. We apply Unbiased Teacher with the proposed OOD filtering (DINO) and use the OpenImagev5 as an unlabeled set, and detection performance of some rare objects are degraded due to the scarcity of these objects in OpenImagev5. Note that OpenImagev5 contains 1.7M images, and COCO2017-train has 117k images.

G Label Correspondence between COCO and OpenImage
To construct the baseline trained with ground-truth labels from OpenImage, we manually label the correspondence between 80 classes in MS-COCO and 601 classes in OpenImage. We provide the object correspondence in Table 13. Among 601 classes in OpenImage, 139 GOI classes have matching COCO classes, and the remaining 462 classes do not correspond to any COCO classes. We thus remove the labels of these classes in the training of the supervised baseline.

H Implementation Details
Our implementation is based on the Detectron2 framework. As our framework is built on the Unbiased Teacher [29], we follow its implementation details, including training iterations, threshold, unsupervised loss weight, and other hyper-parameters for a fair comparison. Model Architecture. We experiment on the Faster-RCNN with FPN [26], and ResNet-50 pretrained on ImageNet-1K is used as the feature backbone. For the offline OOD detectors, we consider DINO [2] and VITB [6] as base models.
Training. For the object detectors, we use the SGD optimizer with a momentum rate 0.9 and a learning rate 0.01, and we use a constant learning rate scheduler for COCO-Open and learning rate decay for COCO-additional and COCO-OpenImage. Each batch contains 8 labeled images and 8 unlabeled images for COCO-Open, and 32 labeled images and 32 unlabeled images for COCO-OpenImage and COCO-additional. To fine-tune DINO/VITB models, we randomly select 64 patches from each image and train 10k/20k/40k iterations for 1k/2k/4k labeled images setups of COCO-Open. For COCO-additional and COCO-OpenImage, we also randomly select 64 patches from each image and train for 160k iterations. We follow the prior work [42] to use SGD optimizer with a learning rate 1e − 3 and 5e − 3 for DINO and VIT models. We apply the inverse abstaining confidence as the OOD score due to its robustness to different number of object categories. As for thresholds for confidence thresholding and OOD filtering, we use δ = 0.5 for the confidence thresholding and δ ood = 0.5 for the OOD filtering. Data augmentation. For COCO-Open, We follow the data augmentation used in Unbiased Teacher [29], which applies a random horizontal flip for weak augmentation and randomly adds color jittering, grayscale, Gaussian blur, and cutout patches [4] for the strong augmentation. For COCO-additional and COCO-OpenImage, we additionally consider scale jitter used in SoftTeacher [49] to further improve the performance. Image-level or box-level geometric augmentations, such as rotation, translation, and Mosaic [55], are not used in our method.

I Training of our online and offline frameworks
In the main paper, we present our proposed OSSOD framework integrated with online and offline OOD detectors. We thus present the training details of offline OOD detectors in Alg. 1 and online OOD detectors in Alg. 2. Limitations and future works. While addressing OSSOD, we do not address other issues such as covariate shift and mismatch in object category distributions between datasets. The offline OOD detector is an individual module from the object detector, so it requires more computational resources in the training stage. However, this concern does not exist as we remove the offline OOD detector and only use the object detector in the inference stage. Our key message is that combining an offline OOD detection module and an SSOD method is a simple yet effective solution to address OSSOD tasks. Based on this integrated framework, there will be more advanced techniques for both SSOD and OOD detection methods, which can potentially improve the performance on OSSOD tasks.