{'fraction_non_alphanumeric': 0.03656099050580201, 'fraction_numerical': 0.012075953583920938, 'mean_word_length': 4.581499845057329, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 3, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
More Practical Scenario of Open-set Object Detection: Open at Category Level and Closed at Super-category Level

Open-set object detection (OSOD) has recently attracted considerable attention. It is to detect unknown objects while correctly detecting/classifying known objects. We ﬁrst point out that the scenario of OSOD considered in recent studies, which considers an unlimited variety of unknown objects similar to open-set recognition (OSR), has a fundamental issue. That is, we cannot determine what to detect and what not for such unlimited unknown objects, which is necessary for detection tasks. This issue leads to difﬁculty with the evaluation of methods’ performance on unknown object detection. We then introduce a novel scenario of OSOD, which deals with only unknown objects that share the super-category with known objects. It has many real-world applications, e.g., detecting an increasing number of ﬁne-grained objects. This new setting is free from the above issue and evaluation difﬁculty. Moreover, it makes detecting unknown objects more realistic owing to the visual similarity between known and unknown objects. We show through experimental results that a simple method based on the uncertainty of class prediction from standard detectors outperforms the current state-of-the-art OSOD methods tested in the previous setting.


Introduction
Open-set object detection (OSOD) is the problem of detecting unknown objects while correctly detecting known objects. Roughly speaking, known objects are the class of objects detectors have seen, and unknown objects are those they have not seen at training time. Previous studies strictly follow open-set recognition (OSR) [27,2] to formalize OSOD [23,6,17,14,29,15,36]. Specifically, they consider the scenario where arbitrary objects are potentially treated as unknown objects. As a result, detectors must detect unknown objects conceptually and visually dissimilar to known objects. This scenario, which we call OSOD-II in this paper, is reflected in the standard benchmark test (Table   1) employed in the previous studies.
In this paper, we first point out this scenario has a fundamental issue leading to a dilemma: we must determine what to detect and what not for unknown objects. Object detectors detect only objects of interest; no object detector detects any "objects." In the ordinary case of closedset object detection, we specify the objects' categories to detect and provide their training data. It is impossible to even conceptually specify what to detect for unknown objects that can be anything. This dilemma emerges because we treat OSOD equally to OSR. It also causes a practical issue: evaluating the accuracy of unknown object detection is hard. Previous studies employ A-OSE [23] and WI [6] as the primary metrics for evaluating OSOD performance. However, these metrics are only partial measures of the accuracy of known object detection; more precisely, they measure the frequency of erroneous detections of known objects as unknowns. They are insufficient for evaluating unknown detection accuracy.
We then introduce a more practical scenario of OSOD, which we name OSOD-III. The previous scenario (i.e., OSOD-II) may not have many practical applications. More practical examples of OSOD are: • For a smartphone app detecting and recognizing animal species, we want the app to detect new animal species that the app's detector has not learned while running.
• For an ADAS (advanced driver-assistance system) detecting traffic signs, we want the system to detect unseen traffic signs in remote areas with insufficient training data.
In both cases, the detection of unknown objects is notified to a higher-level entity, e.g., the system administrators. They may want to update the detectors by collecting training data and retraining them, or they want to analyze the detector's behaviors in an open-set environment. Our new scenario shares these use cases with the previous scenarios. The difference is in the definition of unknown objects. Specifically, we consider only unknown objects belonging to the same super-categories, i.e., animals and traffic signs. The previous scenario considers known and unknown objects that differ at the super-category level.
This difference resolves the dilemma the previous scenario suffers from, i.e., judging what to detect and what not for unknown objects. As a result, this enables the precise evaluation of unknown detection. Object detection has the inherent trade-off between precision and recall. The standard metric for object detection is AP (average precision), i.e., an AUC (area under the curve) metric independent of the operating point of detectors. We can use AP not only for known object detection but for unknown detection. Note that the previous studies have only used A-OSE/WI and have not employed AP.
Another advantage of the new scenario is that detecting unknown objects becomes more realistic. As detectors need only to detect unknown classes within the same super-category as the known objects, we may expect that the known and unknown objects share a visual similarity, helping detectors to detect unknowns. We show through experimental results that a simple method that classifies known and unknown objects using a naïve metric of classification uncertainty works well; it even outperforms existing methods with more complicated mechanisms for detecting unknown objects, which are developed for and evaluated on the previous OSOD scenario.

Open-Set Recognition
For the safe deployment of neural networks, open-set recognition (OSR) has attracted much attention. The task of OSR is to accurately classify known classes and simultaneously detect unseen classes as unknown. Scheirer et al. [27] first formalized the problem of OSR, and many following studies have been conducted so far [2,12,30,25,28,32].
The work of Bendale and Boult [2] is the first to apply deep neural networks to OSR. They use outputs from the penultimate layer of a network to calibrate its prediction scores. Several studies found generative models are effective for OSR, where unseen-class images are synthesized and used for training. Generative Openmax [12] and a counterfactual method [24] use a generative adversarial network [13] to synthesize unseen-class images that are dissimilar to seen-class images. OpenGAN [18] improves the performance of OSR by generating unseen-class samples in the latent space. Another line of the OSR studies focus on a reconstruction-based method using latent features [34,33], class conditional auto-encoder [25], and conditional gaussian distributions [30].

Open-Set Object Detection
Recently, increasing attention is being paid to OSOD. Although it has not been clarified in previous studies, there are mainly two different scenarios in the OSOD study.
One scenario, which we call OSOD-I in this paper, is that we want to detect every known-class object instance in an image while we avoid misclassifying any unseen object instance into known-classes [23,6]. Miller et al. [23] first utilize dropout [11] to estimate the uncertainty of detector's prediction and use it to avoid erroneous detections of unseen-class objects. Dhamija et al. [6] investigate how modern CNN detectors behave in an open-set environment and reveal that the detectors detect unseen objects as known objects with a high confidence score. This scenario does not care the accuracy of detecting unknown objects; the interest lays in the accuracy of detecting known objects. Hence, the researchers have employed A-OSE [23] and WI [6] as the primary metrics to measure the accuracy of detecting known objects. These metrics are designed to measure how frequently a detector wrongly detects and classifies unknown objects as known objects.
The other scenario, we call OSOD-II, is that we want to detect every object instance in an image and then classify it into the correct class if it is a known-class object and identify it as unknown otherwise. This is usually considered a part of open-world object detection (OWOD) [17,14,15,35,29]. This scenario cares the accuracy of detecting unknown objects since it considers updating the detector by collecting unknown classes and using them for retraining. Joseph et al. [17] first introduce the concept of OWOD and establish the benchmark test. Many following studies strictly have followed this benchmark and proposed methods for OSOD. OW-DETR [14] introduces a transformer-based detector (i.e., DETR [3,37]) for OWOD and improves the performance. Han et al. [15] pay attention to the fact that unknown classes are distributed in lowdensity regions in the latent space. They then performed contrastive learning to encourage intra-class compactness and inter-class separation of known classes, leading to the performance gain. Similarly, Du et al. [7] synthesize virtual unseen samples from the decision boundaries of gaussian distributions for each known category.
We basically consider the latter scenario, but find this scenario has an underlying issue that evaluating the accuracy of detecting unknown objects is hard. This is attributed to the benchmark setup, where any objects can be unknown. Thus, we introduce a new scenario of OSOD, which we call OSOD-III. In this scenario, we can rigidly define what to detect and what not for unknown objects, enabling the precise evaluation of unknown detection (i.e., AP for unknown detection). To the authors' knowledge, there is no study employing exactly the same setup. The study of Singh et al. [29] is similar to ours in that they treat unknown ob-jects as those within the relevant categories to the known objects. However, they use A-OSE and WI for evaluation, which is insufficient for evaluating the accuracy of detecting unknown objects. Moreover, we show that a simple naive method works well in the new setting; it even outperforms existing methods having more complicated mechanisms.

Formalizing the Problem
We first formalize the problem of open-set object detection (OSOD). There are two different problems referred to as OSOD without clarification in previous studies. We use the names of OSOD-I and -II to distinguish the two, which are defined as follows.
OSOD-I We want to detect every known-class object instance in an image while we avoid detecting any unknownclass object instance and misclassifying it into known classes.
OSOD-II We want to detect every object instance that should be detected in an image and then classify it into the correct class if it is a known-class object and identify it as such (i.e., unknown) otherwise.
OSOD-I and -II both consider the situation where we apply a closed-set object detector (i.e., a detector trained on closed-set object classes) in an open-set environment, where the detector can encounter objects of unknown class 1 . The difference is whether or not we want to detect unseen-class objects. OSOD-I does not care their detection; it is only interested in the accuracy of detecting known objects. This is considered in [6,23]. On the other hand, OSOD-II cares the detection of unseen-class objects, and thus their detection accuracy matters. This is usually considered as a part of OWOD [17,14,15,35,29,36].

A Fundamental Issue with OSOD-II
Recent studies considering OSOD-II follow the study [17] introducing OWOD for the formalization, which regards it as a generalization of OSR. That is, unknown means 'anything but known.' In other words, any arbitrary objects can be considered unknown. This is reflected in the experimental settings employed in these studies. Table 1 shows the setting, which treats the 20 object classes of PASCAL VOC [8] as knowns and non-overlapping 60 classes from 80 of COCO [20] as unknowns. This class split indicates the basic assumption that there is no relation between known and unknown objects.
However, considering OSOD-II is a detection task, this formalization has an issue since detectors need to detect only objects that should be detected, regardless of whether they are known or unknown. It is the primary problem of object detection to judge whether or not something should be detected. What should not be detected include those belonging to the background and irrelevant (i.e., out-ofinterest) object classes. Detectors learn to do this classification, which is natural when we consider only a closed set of object classes; what to detect is clearly specified (at a conceptual level) in that case. However, this is not the case when we want to detect also unknown objects, which can be any arbitrary ones. It is because we do not have a method to specify what to detect and what not for any arbitrary objects.
A naive solution to this dilemma is to detect any objects as long as they are 'objects.' However, it is not that simple; it is hard to define what is an object. Figure 1 provides examples from COCO images showing the difficulty. COCO covers only 80 object classes (shown in red rectangles in the images), and many more objects are in different scenes (shown in blue rectangles). We will eventually need to annotate every object instance, which is not practical. Moreover, it is sometimes subjective to determine what constitutes individual "objects." For instance, a car consists of multiple parts, such as wheels, side mirrors, and headlights, which we may want to treat as "objects" depending on applications. This difficulty is well recognized in the prior studies of open-world detection [17,14] and zero-shot detection [1,22].

Metrics for Measuring OSOD Performance
The previous studies of OSOD employ two metrics for evaluating methods' performance, i.e., absolute open-set error (A-OSE) [23] and wilderness impact (WI) [6]. These metrics are designed sorely for OSOD-I; they evaluate the accuracy of detecting known objects in open-set environments. Precisely, they measure how frequently a detector wrongly detects and misclassifies unknown objects as known classes.
Nevertheless, previous studies have employed A-OSE and WI as primary performance metrics of OSOD-II. We point out that these are pretty insufficient metrics for Table 1. The class split employed in the standard benchmark test commonly used in recent studies of OSOD [17,14,15,35,36]. Split 1 consists of 20 PASCAL VOC classes. Split 2, 3, and 4 consists of all the classes of COCO super-categories but those of Split 1. A typical setting is to use Split 1 as known categories and Split 2-4 as unknown categories. Note the dissimilarity between the known and unknown categories.
A-OSE and WI cannot evaluate this accuracy since these metrics merely evaluate how frequently detectors wrongly detect/classify unknown objects as known, as mentioned above. A-OSE/WI evaluate only one type of error, i.e., detecting unknown as known, and ignore the other type of error, detecting known as unknown.
We also point out that A-OSE and WI are not flawless even as an OSOD-I performance metric. That is, they merely measure the detectors' performance at a single operating point; they cannot take the precision-recall tradeoff into account, which is the fundamental nature of detection. Specifically, previous studies [17] report A-OSE values for bounding boxes with confidence score ≥ 0.05 2 . As for WI, previous studies [17,14,15,29] choose the operating point of recall = 0.8. We may say that the confidence threshold = 0.05 for A-OSE is not a realistic operating point of object detectors. While that for WI may be better, it is still insufficient since the final choice must be left to applications.
Figures 2(a) and (b) show the profiles of A-OSE and WI, respectively, over the possible operating points of several OSOD-II detectors that will be compared in our experiments. It is seen that the ranking of the methods significantly depends on the choice of confidence threshold. Thus, we can say we should be cautious to use these metrics to compare OSOD methods.
As described above, these metrics are insufficient for evaluating OSOD-II methods since i) they only measure OSOD-I performance, e.g., only one of the two error types; and ii) they are metrics at a single operating point. To measure OSOD-II detection performance, rigorously the accuracy of detecting unknown objects, we should use mAP. However, while all the previous studies report mAP of known object detection, no study reports mAP of unknown object detection, with a single exception [15].
We can raise two possible explanations. One is that OSOD-II is formalized as an extension of OSR as above, where anything can be unknown. If we strictly follow this discipline, we cannot provide ground truths for the detection of unknowns, making the computation of mAP impossible. However, the previous studies specify unknowns in 2 This is not clearly stated in the literature but can be confirmed with the public source code in GitHub repositories, e.g., https://github. com/JosephKJ/OWOD. their experiments as in Table 1 3 . Thus, we can calculate mAP for those specified without an additional effort. Why do previous studies not do this? It may arguably be because the computed mAP tends to be too low. Indeed, detecting unknown objects specified as in Table 1 is quite hard.

A-OSE and WI
A-OSE is the number of predicted boxes wrongly classified as known categories (that are unknown objects in reality) [23]. WI measures the ratio of the number of erroneous detections of unknowns as knowns (i.e., A-OSE) to the total number of detections of known instances, given by where P K indicates the precision measured in the close-set setting, and P K∪U is that measured in the open-set setting.

Yet Another OSOD Scenario
This section introduces another setting of OSOD, which we name OSOD-III. Although it has been overlooked in previous studies, this new setting considers a situation we frequently encounter in practice. It has good characteristics that allow us to find practical solutions that are not possible with OSOD-I/II.

OSOD-III: Open at Class Level and Closed at Super-category level
Consider building a smartphone app that detects and classifies animal species. It is unrealistic to deal with all  Figure 3. Illustration of OSOD-II and OSOD-III. OSOD-II assumes unknown objects to be arbitrarily dissimilar to known objects (top row). OSOD-III assumes unknown objects to belong to the same super-category as the known objects; unknown and known classes share the conceptual and visual similarities (bottom row).
animal species from the beginning since there are too many classes. Thus, consider a strategy to start the app's service with a limited number of classes; after its deployment, we want to add new classes by detecting unseen animal categories. To do this, we must design the detector to detect unseen animals accurately while correctly detecting known animals. After detecting unseen animals, we may collect their training data and retrain the detector using them. There will be many similar cases in real-world applications.
Such a problem falls in the OSOD framework. Similar to OSOD-II, we want to detect unknown, novel animals. However, unlike OSOD-II, it is unnecessary to consider arbitrary objects as detection targets. In brief, we consider only animal classes; our detector does not need to detect any non-animal object, irrespective of whether it is known or unknown. In other words, we consider the set of object classes closed at the super-category level (i.e., animals) and open at the individual class level under the super-category.
We call this setting OSOD-III. Figure 3 illustrates the differences between OSOD-II and -III. This new setting is formally stated as follows: OSOD-III Assume we are given a closed set of object classes belonging to a single super-category. Then, we want to detect and classify objects of these known classes correctly and to detect every unknown class object belonging to the same super-category and classify it as such.
It is noted that there may be multiple super-categories instead of a single. In that case, we need only consider the union of the super-categories. For the sake of simplicity, we only consider the case of a single super-category in what follows.

Advantages of OSOD-III
While the applicability of OSOD-III is narrower than OSOD-II due to the restriction on super-categories, OSOD-III has two good properties OSOD-II does not have 4 .
One is that OSOD-III is free from the fundamental difficulty of OSOD-II, the dilemma of determining what unknown objects to detect and what to not. As a result, OSOD-III no longer suffers from the evaluation difficulty. Indeed, the judgment is clear with OSOD-III; unknowns belonging to the known super-category should be detected, and all other unknowns should not. Then, this clear separation enables the computation of average precision (AP) for unknown objects, allowing a practical comparison of different methods.
The other is that detecting unknowns will become easier owing to the similarity between known and unknown categories. In OSOD-II, unknown objects can be arbitrarily dissimilar from known objects. In OSOD-III, known and unknown objects share their super-category, leading to the conceptual and visual similarity between them. It should be noted here that what we regard as a super-category is arbitrary; there is no mathematical definition. However, as far as we consider reasonable class hierarchy as in Word-Net/ImageNet [9,5] , we can say they will share visual similarities. Table 2. The details of the four splits of categories. We treat one of the four as a known set and the other three as a unknown set. Thus, there are four cases of known/unknown splits, for each of which we report the detection performance in Table 3.   Animal  Vehicle  Split1  Split2  Split3  Split4  Split1  Split2  Split3 Split4  num of known categories  24  24  24  24  6  6  6  6  train images  44379 38914 39039 18478  43270 26860  3900  6300  validation images  1104  2353  1248  849  1370  503  178  322  test images 15609 6991

A Simple Baseline Method
As mentioned above, detecting unknowns in OSOD-III is not so hard as in OSOD-II. We will show later in Sec. 5.3 that a simple, naive method works well, which merely uses the class scores that standard detectors predict for each bounding box. It relies on an expectation that unknownclass inputs should result in uncertain predictions of classes. Thus we look at the prediction uncertainty to judge if the input belongs to known/unknown classes.
Several methods are empirically known to be effective in quantifying the uncertainty, such as the maximum of the softmax values (i.e., the largest score) and the entropy. We found in our preliminary experiments that the ratio of the top-1 and top-2 scores works slightly better. Specifically, we calculate the ratio (top-1)/(top-2) for each candidate bounding box and compare it with a pre-defined threshold γ; we regard the input as known if it is higher than γ and as unknown otherwise. If a bounding box is judged known, we use the highest class score as its confidence score as usual; if judged unknown, we use the sum of the top three class scores as the box's confidence score.
This method can be used with any base object detector. In our experiments, we employ two popular detectors, FCOS [31] and Faster RCNN [26]. FCOS is a single-stage, anchor-free detector that calculates each class's score independently with a logistic sigmoid function. Faster RCNN is a two-stage detector and calculates the class scores with the softmax function.

Dataset
We use Open Images dataset v6 [19] for our experiments. It contains 1.9M images of diverse objects (601 categories) with 15.9M bounding box annotations. It also provides the hierarchy of object categories in a tree structure, whose nodes represent super-categories and leaves represent individual object categories, e.g., a leaf, Polar Bear, has a parent node, Carnivore, whose parent node is Mammal.

Experimental Scenarios
We choose two super-categories, Animal and Vehicle in our experiments. There are 96 and 24 categories in the Animal and Vehicle super-category, as in Table 2. Let S be the set of categories for each. Considering statistical instability due to the choice of known/unknown classes, we randomly split S into four subsets. We select one of them for the knownclass set and use the union of the other three subsets for the unknown-class set. As there are four choices of the knownclass set, we report the results for each of them; see the supplementary material for more details.
After selecting known/unknown categories, we choose images for training, validation, and testing as follows. For the training set of images, we choose the images containing at least one instance (i.e., bounding box) with a known category from the original training split of the dataset. For the validation and test sets, we choose the images containing at least one instance with a known category or an unknown category from the original validation and test splits, respectively.
The training images thus obtained may contain not only known objects but also unknown objects and irrelevant objects. Some irrelevant objects may have bounding box annotations. We keep only annotations for known objects and remove all other annotations (including unknowns 5 ). Thus, those removed objects will be potentially treated as the 'background' class. For the test and validation images, we keep the annotations for known and unknown objects, and remove those for all other irrelevant objects.
Note that the validation images are used by the compared methods (i.e., ORE [17] and VOS [7] in this paper) and not used by the other methods including our baseline. ORE and VOS rely on the availability of (example) unknown objects, which may be regarded as leakage especially in the setting of OSOD-III.

Evaluation
It is doubtless that average precision (AP) is the standard metric for evaluating object detection [10,8]. Owing to the mentioned properties of OSOD-III, we can use AP to evalu- Table 3. Detection accuracy of known (AP known ) and unknown objects (AP unk ) of different methods in the OSOD-III setting. 'Split-n' indicates that Split-n in Table 2  ate the detection of unknown objects, unlike OSOD-II. Following the evaluation procedure of the standard object detection, we report AP over IOU in the range [0.50, 0.95] also for unknown detection.

Compared Methods
We consider three existing OSOD methods [17,7,15] that are designed to detect unknown objects in addition to known objects. We summarize these methods and their configurations in our experiments below. We compare them and the simple baseline method explained in Sec. 4.3.

ORE (Open World Object Detector)
ORE is the method proposed in [17]. It is originally designed for OWOD and thus is capable not only of detecting unknown objects but also of incremental learning. We simply omit the latter capability and use the former as an open-set object detector. It employs an energy based method to judge known/unknown; using the validation set including unknown object annotations, it models the energy distributions for known and unknown objects. ORE provides a detection score for unknown objects, which we use to compute AP. We follow the paper [17] for the detector architecture, i.e., Faster RCNN with a ResNet50 backbone [16].
VOS (Virtual Outlier Synthesis) VOS detects unknown objects by treating them as out-of-distribution (OOD) based on an energy-based method [7] . Specifically, it estimates an energy value for each detected instance and judges whether the instance is known or unknown by comparing its energy with a threshold. Following the original study [7], we choose the threshold so that 95% of known object instances are classified correctly as such with the validation set. VOS does not provide a detection score for unknown objects. As we need a score to calculate AP, we use the energy value for this purpose. We also follow the paper for the base detector, Faster RCNN [26] with ResNet50-FPN backbone [21].
OpenDet (Open-set Detector) OpenDet [15] is the current state-of-the-art 6 on the standard benchmark using PASCAL VOC/COCO objects shown in Table 1. It provides a detection score also for unknown objects, which we utilize it to calculate AP. We employ the authors' implementation employing Faster RCNN based on ResNet50-FPN.

Simple Baselines
As explained in Sec. 4.3, our simple baseline needs a hyperparameter γ to classify known/unknown objects. We set γ = 4 for FCOS and γ = 15 for Faster RCNN; the difference is caused by the design of output layers, i.e., logistic vs. softmax. We report the sensitivity to the choice of γ (i.e., the results with different values) in the supplementary. We use ResNet50-FPN as the backbone for both FCOS and Faster RCNN. Table 3 shows the results for the two scenarios, where the super-category is "Animal" and "Vehicle", respectively. It shows mAP for the known-class objects, denoted by AP known , and AP for the unknown objects, denoted by AP unk , for each of the four splits of known/unknown classes, and their averages (with standard deviations) as well.

Results
We can see from the table that the compared methods attain similar AP known except the FCOS-based baseline. It shows noticeably lower accuracy, which is attributable to the diffrence in the base detector. Then, comparing AP unk , we can observe that the two baseline methods outperform the other methods despite the fact that they have specialized mechanisms for OSOD. Specifically, the one with the same detector base model (i.e., Faster RCNN) achieves slightly better performance than OpenDet, reported as the state-ofthe-art in the OSOD-II setting. The FCOS-based one yields further better performance.  Figure 4. Visualized output for each models. Our simple baselines (FCOS and Faster RCNN) detect unknown categories as comparably well as OpenDet, which is the state-of-the-art method for OSOD. Red boxes are detections/ground truth labels for unknown categories, and blue boxes are that of known categories. "Unk" in the image is equivalent to "unknown". We use confidence threshold = 0.3 for the visualization. Figure 4 shows several selected examples of detection results by OpenDet and the two baselines. We can see that the three methods behave mostly similarly; some wrongly detect a known object as unknown or a unknown object as known. However, it is important to note that these error types are properly taken into account by AP unk .

Conclusion
In this paper, we have reconsidered open-set object detection (OSOD). Classifying the current settings of OSOD into two (i.e., OSOD-I and -II), we first pointed out that OSOD-II has a fundamental issue. Specifically, it suffers from a dilemma of determining what to detect and what not for unknown objects that we cannot identify by all means beforehand. This dilemma leads to difficulty with the evaluation of the accuracy of detecting unknown objects. Although we should use the standard metric, AP, for the evaluation, the previous studies employ A-OSE and WI. We explained these metrics are pretty insufficient for evaluating unknown detection as they are originally designed for OSOD-I, i.e., detecting known objects as accurately as possible in open-set environments.
We then introduced a new scenario of OSOD, named OSOD-III. It considers the detection of unknown objects that share the same super-category as the known objects. This scenario has broad applicability in real-world problems. Moreover, it resolves the issue of OSOD-II. We no longer suffer from the dilemma of determining what to detect and what not for unknown objects. It also enables to use AP for the evaluation of unknown object detection. Furthermore, detecting unknown objects becomes more realistic owing to a visual similarity between the known and unknown objects. We showed through experimental results that a simple approach of using classification uncertainty to judge whether the candidate instance is known or unknown outperforms the state-of-the-art methods with more complicated mechanisms developed for and tested on OSOD-II. Table 4 shows the details of the category splits for the two super-categories, "Animal" and "Vehicle," used in our experiments, whose summary is provided in Table 2 of the main paper.

A.2. Training
For all the compared methods, we train the network for 12 epochs using the SGD optimizer with the batch size of 16. We use the initial learning rate of 2.0 × 10 −2 with momentum = 0.9 and weight decay = 1.0 × 10 −4 . We use the publicly available source code for ORE 7 [17], VOS 8 [7], and OpenDet 9 [15]. We use mmdetection 10 [4] to implement/test FCOS [31] and Faster RCNN [26] for our baseline method.

B. Results of A-OSE and WI
In this study, we use the average precision for unknown object detection, denoted by AP unk , as a primary metric to evaluate OSOD methods, as reported in Table 3. For the readers' information, we report here absolute open-set error (A-OSE) and wilderness impact (WI), the metrics widely used in previous studies. Table 5 shows those for the compared methods on the same test data as Table 3.
Note that A-OSE and WI measure only detectors' performance of known object detection and thus are insufficient as a metric of OSOD, as explained in Sec.3.3. Note also that these metrics evaluate detectors' performance at a single operating point; Table 5 provides only a snapshot of Fig. 2, which shows the comprehensive behaviors of these detectors over different operating points. Table 5 shows the results at the operating points chosen in the previous studies, i.e., confidence score > 0.05 for A-OSE and the recall (of known object detection) = 0.8 for WI, respectively. We can observe from the table that our baseline method with Faster-RCNN yields better values than the state-of-the-art Open-Det. On the other hand, the one with FCOS yields worse values, which may be attributable to the specific choice of the operating point; see Fig. 2 also.

C. Effect of Hyperparameters for the Baseline Method
Our baseline method uses the ratio of the top two class scores to classify an object into known and unknown, where we use a hyperparameter γ as the threshold. In the experiments reported in the main paper, we manually chose it for 7 https://github.com/JosephKJ/OWOD.git 8 https://github.com/deeplearning-wisc/vos.git 9 https://github.com/csuhan/opendet2.git 10 https://github.com/open-mmlab/mmdetection.git each of the variants with FCOS and Faster-RCNN in our experiments. That is, we did not follow the standard procedure using validation data since, considering the nature of OSOD, it may not be appropriate to assume samples labeled as 'unknown' to be available. So, we report the results with different values of γ to see the sensitivity to its choice here. Table 6 and 7 show the results for the baselines with FCOS and Faster RCNN, respectively. We can observe that overall, while the results vary depending on the choice of γ (and also T with Faster RCNN), AP known and AP unk are not extremely sensitive to their choice. There is a trade-off between AP known and AP unk , since small γ's make the detectors overlook unknown objects while large γ's make the detectors overlook known objects. A good balance of AP known and AP unk is achieved at γ ∈ [3.0, 5.0] for FCOS and γ ∈ [5.0, 15.0] for Faster RCNN, respectively. Setting a large temperature T (> 1) with Faster RCNN helps improve AP unk , at a small sacrifice of AP known ; too large T damages performance. Note that we set γ = 4.0 for FCOS and γ = 15.0 and T = 1 for Faster RCNN in the experiments reported in the main paper. Figure 5 shows more detection results of the compared methods. We only show the bounding boxes with confidence score > 0.3. These show that our baselines and OpenDet perform equally well in detecting unknown objects. ORE and VOS tend to classify unknown objects wrongly into a known category.   Table 7. Results of the baseline with Faster RCNN with different combinations of γ and temperature T . The numbers stand for AP known / AP unk / WI.