{'fraction_non_alphanumeric': 0.02880052364588447, 'fraction_numerical': 0.012354770086728849, 'mean_word_length': 4.9854717597127, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Contrastive Object Detection Using Knowledge Graph Embeddings

Object recognition for the most part has been approached as a one-hot problem that treats classes to be discrete and unrelated. Each image region has to be assigned to one member of a set of objects, including a background class, disregarding any similarities in the object types. In this work, we compare the error statistics of the class embeddings learned from a one-hot approach with semantically structured embeddings from natural language processing or knowledge graphs that are widely applied in open world object detection. Extensive experimental results on multiple knowledge-embeddings as well as distance metrics indicate that knowledge-based class representations result in more semantically grounded misclassifications while performing on par compared to one-hot methods on the challenging COCO and Cityscapes object detection benchmarks. We generalize our findings to multiple object detection architectures by proposing a knowledge-embedded design for keypoint-based and transformer-based object detection architectures.


Introduction
Object detection is the task of localizing and classifying objects in an image. Deep object detection originates from the two-stage approach [1,2,3] which first searches for anchor points in the image that potentially contain objects, followed by classifying and regressing the coordinates for each proposal region in isolation. The Faster R-CNN approach [1] achieves an average precision of 21.9% on the challenging COCO 2017 benchmark [4] for object detection in everyday situations. Its dependence on a heuristic suppression of multiple detections for the same object is a commonly addressed design flaw [5]. In an effort to make the network end-to-end trainable, recent architectures model the relationship between object proposals by formulating object detection as a keypoint detection [6,7] or a direct-set estimation problem [8,9,10]. Such novel architectures along with stronger backbones [11,12] and multiscale feature learning techniques [13,14,15] have doubled the average precision Figure 1: We incorporate class prototypes such as GloVE [16] from natural language processing, into the prediction head of common object detectors. The resulting feature space is structured by the context from non-visual knowledge sources such that similar concepts are in proximity, which we show yields more semantically-grounded predictions. on the COCO detection benchmark to 54.0% [10].
These astonishing benchmark performances have grown at the expense of a set-based classification formulation, i.e., the classes are regarded as discrete and unrelated. This assumption allows such methods to use a multinomial logistic loss [1,17,6] to boost the classification accuracy by maximizing the inter-class distances [18]. However, this forces the model to treat each class prototype out-of-context from what it has learned about the appearance of the other classes. As a result, the training is prone to inherit bias through classimbalance [19,20] or co-occurrence statistics [21] from the training data, and neglects synergies in the real world, where there exist semantic similarities between classes. This closed-set formulation further requires drastic changes of the class prototypes to be learned whenever the set of classes is extended [22,23], resulting in catastrophic forgetting of previously learned classes when the training is performed incrementally [23,24].
In zero-shot learning, that aims to detect classes that were excluded from the training set, the parametric class proto-types are replaced with a mapping from visual to a semantic embedding space, e.g. word embeddings generated from text corpora, which describe semantic context for a broad set of object classes. We extend this approach to keypointbased and transformer-based object detectors and evaluate various knowledge embeddings as well as loss functions on the COCO 2017 [4] and Cityscapes [25] datasets. Additionally, we analyze the classification error distributions of our proposed KGE architectures with baseline methods of set-based classification and show that KGE predictions tend to be more semantically consistent with the groundtruth.
In summary, our main contributions in this work are: • a knowledge embedding formulation for two-stage, keypoint-based and transformer-based multi-object detection architectures. • a quantitative ablation study on the COCO 2017 over different distance metrics and knowledge types. • an error overlap analysis between one-hot and embedding-based classifications.

Related Work
This work relates to the research areas of object detection architectures, especially their classification losses and the usage of commonsense knowledge in computer vision. In this section, we review the relevant works in these areas.
Object Detection: Two-stage methods [1,2,17] rely on a Region Proposal Network (RPN) to first filter image regions containing objects from a set of search anchors. In a second stage, a Region of Interest (ROI) head extracts visual features from which it regresses enclosing bounding boxes as well as predicts a class label from a multi-set of classes for each object proposal in isolation. Classification is then learned using a cross-entropy loss across the confidence scores for each class, including a background class, where the class labels are found by matching the groundtruth bounding boxes to object proposal boxes using only the intersection over union (IOU). This formulation is vulnerable to imbalanced class distributions in the training data, therefore a focal loss [19] down-weights well-classified examples in the cross-entropy computation to focus the training on hard examples.
Anchor-less detectors such as keypoint-based or centerbased methods [26,6,7] embody an end-to-end approach that directly predicts objects as class-heatmaps. They use a logistic regression loss for training using multi-variate Gaussian distributions each centered at a groundtruth box. The classification stage has a global receptive field, which enables implicit modeling of context between detections. Transformer-based methods [8,27,10] currently lead the COCO 2017 [4] object detection benchmark. These methods detect objects using cross-attention between the learned object queries and visual embedding keys, as well as self-attention between object queries to capture their interrelations in the scene context. DETR [8] builds upon ResNet as a feature extractor and a transformer encoder to compute self-attention among all pixels of the feature map. However, the quadratic complexity w.r.t. the input resolution makes it infeasible for high-resolution feature maps, leading to poor performance on small objects. More efficient querying strategies, e.g., DeformableDETR [9], was proposed that also support multiscale feature maps.
Knowledge-based embeddings: In zero-shot classification [28,29] and recognition [30,31,32,33], word embeddings commonly replace learnable class prototypes to transfer from training classes to unseen classes using inherit semantic relationships extracted from text corpora. Commonly used word embeddings are GloVe vectors [29,30] and word2vec embeddings [31,34,33,35], however embeddings learnt from image-text pairs using the CLIP [36] achieve the best zero-shot performance so far [32]. Rahman et al. [31] argue that a single word embedding per class is insufficient to model the visual-semantic relationships and propose to learn class representations of weighted word embeddings of synonyms and related terms [31]. Nevertheless, pure text embeddings perform consistently best for training classes [30,31,32,33,35,32] in object detection. The projection from visual to semantic space is done by a linear layer [30,37,35], a single [31,34,33] or two-layer MLP [32], and learned with a max-margin losses [30,31,38,37], softplus-margin focal loss [35], or crossentropy loss [33,32]. Zhang et. al. [39] suggests to rather a mapping from semantic to visual space to benefit to alleviate the hubness problem in semantic spaces, however, our analysis of COCO 2017 class vectors suggested that the hubness phenomena does not have a noticeable impact on the COCO 2017 class embedding space.
Another fundamental design choice in zero-shot object detection is the background class representation. The majority of works rely on an explicit class prototype that is either learned [33,32], computed as the mean of all class vectors [31,34,37], or represented by multiple background word embeddings [30]. However, Li et al. [35] noted that an explicit representation can cause confusions with unknown classes and therefore represented background by a distance threshold w.r.t. each class prototype.
In this work, we analyze the usage of semantic relations between embeddings from the error distribution point of view and extend it to keypoint-based and transformer-based object detection architectures. We compare contrastive loss with margin-based loss functions and ablate multiple embedding sources to strengthen our analysis.

Technical Approach
Closed-set object detection architectures [1,6,8,40] share a classification head R D → R C design incorporat-  . The CenterNet (a) encodes the image into a downscaled feature map from which it regresses a per-pixel class embedding vector, bounding box width and height, as well as centerpoint offset. Object localization is performed by computing distance fields to each class prototype. For all the distances lower than a background threshold, we perform a local minima search in an 8-connected neighborhood to locate object center points. DETR (b) employs a transformer encoder and decoder with object queries as keys to detect objects. The decoded values are fed into an MLP that outputs bounding box coordinates as well as an embedding vector that is used for nearest neighbor classification given semantic class prototypes.
ing a linear output layer, that maps the feature vector into a vector of class scores of the form where W ∈ R D×C is the parameter matrix of the linear layer, which effectively is composed of C learnable class prototypes. These parameters are learned with a one-hot loss formulation, incentivizing the resulting class prototypes to be pairwise orthogonal. We replace learnable class prototypes with fixed object type embeddings that lend structure from external knowledge domains in the closed-set setting. Essentially, this entails replacing the direct estimation of classification scores in object detection with a regression that maps image regions into a shared space of visual features and semantic embedding vectors, where classification is done by a nearest neighbor search. In the remainder of this section, we present the possible class prototype choices in Sec. 3.1, distance metrics in Sec. 3.2, and describe the incorporation into two-stage detectors, keypoint-based method, and transformer-based architectures in Sec. 3.3.

Object-Type Embeddings
Instead of handcrafting the regression target vector or learning them from the training data, we incorporate object type embeddings from other knowledge modalities as follows: • GloVe word embeddings [41] build on the descriptive nature of human language learned from a web-based text corpus. Human language reflects a variety of dependencies between objects -including causal, compositional, as well as spatial interrelations-which is expected to reflect in the embeddings. • ConceptNet graph embeddings encode the conceptual knowledge between object categories entailed in the ConceptNet knowledge graphs [42]. The embeddings are computed by pointwise mutual information of the adjacency matrix and projected into a 100-D vector space by truncated SVG, as described by Speer et al. [42]. • COCO 2017 graph embeddings reflect the cooccurrence commonsense learned from the COCO 2017 training split. We build a heterogeneous graph from the spatial relations defined by Yatskar et al. [43]: {touches, above, besides, holds, on} and transform the resulting graph into node embeddings for each object prototype with an R-GCN [44].

Distance Metrics for Classification
We analyze nearest neighbor classification between visual embeddings of proposal boxes and object type representation vectors, either by the norm of the difference vector or angular distance. The L k norm of the difference vector between two embedding vectors is given by, where b i ∈ R D denotes the feature vector of the i-th proposal box and t c ∈ R D the class prototype of the c-th object class. The maximum distance bound is given by the triangle inequality as d k,max ≤ |t c | k + |b i | k . In order to have a bound independent of D, we project all embeddings inside the unit sphere as Since the relative contrast in higher dimensions is largest for low values of k [45], we analyze the Manhattan distance where k = 1. We further investigate nearest neighbors in terms of the cosine angle between two embedding vectors.
The distance is computed using the negated cosine similarity as We then interpret the negated distance as a similarity measure, and define the embedding losses as a contrastive loss towards the class embedding vectors given by where c i is the label of the groundtruth box to which the proposal is matched. τ denotes a fixed scaling factor of the similarity vector, commonly referred to as temperature. We follow [46] and fix τ = 0.07. In case of a feature vector b i that is assigned to the background label, we set c = 0 and sim(b i , t 0 ) = 0, as there is no class representation vector that the feature embedding should have close distance too. Please refer to the supplementary material for a comparison to a margin-based loss function.

Generalization to Existing Object Detectors
In this section, we describe the embedding extension to three types of object detectors: two-stage detectors, keypointbased methods, and transformer-based.
Two-stage detectors such as R-CNN based methods [1] pools latent features of each object proposals regions to fixed-size feature vector in the ROI head. These features are then processed in a box regressor and classifier to refine and classify the boxes for each proposal in isolation. The base method is trained with a standard Faster R-CNN loss, including a cross-entropy loss for bounding box classification. For the Faster R-CNN KGE method, we replace the crossentropy classification loss for an embedding loss function, such that the linear layer learns a regression towards fixed class prototype vectors rather than mutually optimizing both. We therefore task the linear output layer to map from the ROI feature dimensionality to the embedding dimensionality D. We replaced the ReLU activation at the box head output layer with a hyperbolic tangent activation function (tanh) to avoid zero-capping the ROI feature vectors before the final linear layer.
Keypoint-based detectors formulates object detections as a keypoint estimation problem. CenterNet [6] uses these keypoints as the center of the bounding boxes. The output stage has a convolutional kernel with filter depth equal to the number of object classes, as well as additional output heads for center point regression, bounding box width, and height, etc. Max-pooling on class-likelihood maps yields a unique anchor for each object under the assumption that no two bounding boxes share a center point. The class-likelihood maps are trained with a focal logistic regression loss, where each groundtruth bounding box is represented by a bivariate normal distribution Y xyc with mean at the groundtruth center point and class-specific variances.
For the KGE formulation, we use the filter depth of the embedding output stage according to the number of embedding dimensions, thus learning an embedding for each pixel in the output map as shown in Fig. 2a. Center point search then requires to compute a distance field of each pixel embedding to the class representation vectors. We formulate the embedding based focal loss as where sim(b i , t c ) is a similarity function, α and β are hyperparameters of the focal loss, and Y xyc is a heatmap where each groundtruth bounding box of class c is represented by a bivariate normal distribution Y xyc at its center point coordinates x and y, scaled such that Y xyc = 1.
Transformer-based methods [8,9] adopt an encoderdecoder architecture to map a ResNet-encoded image to features of a set of object queries. The DETR encoder computes self-attention of pixels in the ResNet output feature map concatenated with a positional embedding. The decoder consists of cross-attention modules that extract features as values, whereby the query elements are of N object queries and the key elements are of the output feature maps from the encoder. These are followed by self-attention modules among object queries. Each feature vector is then independently decoded into prediction box coordinates and class labels with a 3-layer perceptron with ReLU activation [8], as depicted in Fig. 2b. The network is trained with a unique matching between proposal boxes and groundtruth boxes based on a classification cost with the negative log-likelihood of class labels, a generalized IOU loss as well as L1 regression loss on the bounding box coordinates. We propose to replace the class label prediction with a feature embedding regression, and replace the classification cost function in the Hungarian matcher by the negative logarithm of the similarity measure, as described in Sec. 3.2. We further replace the ReLU activation function of the secondlast linear layer with a tanh activation function.

Experimental Evaluation
We evaluate the nearest neighbor classification heads for 2D object detection on the COCO 2017 and Cityscapes benchmarks. We use the mean average precision (AP ) metric averaged over IoU ∈ [0.5 : 0.05 : 0.95] (COCO's standard metric) as the primary evaluation criteria. We also report the AP w where we weight each class by the number of groundtruth instances, the average precision across scales COCO   AP S,M,L and the categories AP cat for completeness. The latter is a novel metric first introduced in this paper, which is based on categorizing the 80 COCO 2017 thing classes into a total of 19 base categories by finding common parent nodes based on WUP similarity in the Google knowledge graph [47]. A true positive classification label hereby defines the correct category rather than class. Please refer to the supplementary material for an overview of object categories. We chose the object detection task over image classification as it additionally requires predicting whether a proposed image region contains a foreground object. Ablation studies are performed on the validation set, and a architecture-level comparison is reported on the COCO 2017 test-dev and Cityscapes val datasets. For object detection algorithms, we compare Faster R-CNN [1] as representative for anchorbased methods, CenterNet [6] as keypoint-based object detector, and DETR [8] as transformer-based representative.

Datasets
COCO is currently the most widely used object detection dataset and benchmark. The data depicts complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentation to aid in precise object localization [4]. We use annotations for 80 "thing" objects types in the 2017 train/val split, with a total of 886,284 labeled instances in 122,266 images. The scenes range from dining table close-ups to complex traffic scenes.
Cityscapes is a large-scale database that focuses on seman-tic understanding of urban street scenes. It provides object instance annotations for 8 classes of around 5000 fine annotated images captured in 50 cities during various months, daytime, and good weather conditions [25]. The scenes are extremely cluttered with many dynamic objects such as pedestrians and cyclists that are often grouped near one another or partially occluded.

Training Protocol
We use the PyTorch [48] framework for implementing all architectures, and we train our models on a system with an Intel Xenon@2.20GHz processor and NVIDIA TITAN RTX GPUs. For comparability, we use a ResNet-50 backbone with weights pre-trained as an image classifier on the ImageNet dataset for all experiments, and the same training configurations as reported in [1] for Faster R-CNN KGE, in [6] for CenterNet KGE, and as in [8] for DETR KGE configurations. We rescale the images to 640 × 480 pixels and use random flipping as well as cropping for augmentation.

Quantitative Results
On the COCO 2017 test set in Tab. 1, we compare our results against baselines from the original methods presented in Sec. 3.3. We expected the baselines methods to outperform our proposed knowledge graph embedded (KGE) prediction heads since they favor one-hot encoded class prototypes which are pairwise orthogonal and therefore maximize inter-class distances. Nevertheless, Tab. 1 shows that the knowledge-based class prototypes in the final classification layer can compete with their baseline configurations in terms of the AP metrics.  Table 3: Ablation study on COCO 2017 validation set results over knowledge embeddings in 100 dimensions using a cossim distance metric and a contrastive loss ( (6)). The class prototypes are described in Sec. 3.1 high accuracy bounding boxes. We attribute this competitive performance of the KGE methods to the small temperature value of τ = 0.07, which was used as a hyperparameter for the contrastive loss function. As Kornblith et al. [18] noted, the temperature controls a tradeoff between the generalizability of penultimate-layer features and the accuracy on the target dataset. We performed a grid search over temperature parameters and found that accuracy drops considerably with larger temperature values, as shown in the supplementary material. Another interesting observation are the large AP category,w scores, where the KGE methods consistently outperform their baselines by greater than 1.5%. These results demonstrate that misclassifications are more often within the same category, in contrast to baseline methods that appear to confuse classes across categories.
The results on the Cityscapes val set are shown in Tab. 2. The CenterNet KGE variant achieves the best precision with AP = 42.2%. We should note, that for this dataset with only few classes that are highly related, the Faster R-CNN KGE methods performs worse than its baseline, mainly due to few accurately predicted bounding boxes as can be seen by the low AP 75 = 34.4.

Ablation Study
In this section, we compare three class prototype representations described in Sec. 3.1 to analyze which aspects of semantic context are most important for the object detection task and evaluate distance metrics used for classification. higher AP and lowest for DETR with +1.02AP . This might reflect the sensitivity to outliers for each method and the separation of the embedding space. In high dimensions, the euclidean distance between two points is inherently large, since the vector becomes increasingly sparser. According to Aggarwal et al. [45], this results in decreasing contrast by the L k -norm distance between embedding vectors with increasing dimensionality, which hurts classification performance. The cosine similarity on the other hand reports the same similarity between each point on a line from the origin to a target point, in principle reserving the scaling dimension to account for different appearances of an object class. Further, the effect of outlier values in a single dimension only enters the distance metric at most linearly.

Misclassification Analysis
We analyze the confusion matrix of each detector on the COCO 2017 validation split. The confusion matrix E ∈ R C×C contains a row for each object class, and each entry represents the counts of predicted labels for this class groundtruth instances. For each groundtruth box, we select the predicted bounding box with the highest confidence score of all predicted boxes with an IOU ≥ 0.8, i.e., each groundtruth box is assigned to at most one prediction.
For comparing error distributions of different object detectors, we compute the Jensen-Shannon distance of corresponding rows in the confusion matrices as where p, q are false positive distributions, m is the pointwise mean of two vectors as m i = pi+qi 2 , and D is the Kullback-Leibler divergence.
A low JS distance implies that the classification stages of the two detectors under comparison produce similar error distributions and vice versa. Fig. 3 shows the JS distances for each object detection algorithm between the baseline method and all embedding-based prediction heads. We note that the error distributions vary noticeably from the baselines methods, however all the embedding-based prediction heads appear to differ by similar magnitudes. This behavior indicates that there is a conceptual difference in the prediction characteristic of the learned (baseline) and knowledge-based class prototypes. The effect is largest for the CenterNet architecture, presumably since the embedding-based formulation affects the classification as well as localization head. To further investigate the origin of this dissimilar behavior, we quantify the inter-category confusions of different object detector architectures in Fig. 4.
Inter-Category Confusion: Fig. 4 shows that all knowledge-embedded class prototypes, except for the COCO 2017 class prototypes, produce lower inter-category confusions compared to their one-hot encoded configurations. This signifies that feature embeddings derived from spatial relations between objects provide insufficient inter-class distances when used as class prototypes. This impairment is especially noticeable for the Faster R-CNN architecture, where class confusion of overlapping objects can also occur during the loss computation. The ConceptNet embeddings result in the lowest counts of inter-category confusion, since its structure is derived from a conceptual knowledge graph similar to the one used for categorization, and its inter-category confusion counts are consistently lower than the base configuration with learnable class prototypes.

Qualitative Results
We further qualitatively evaluate images from the Open-Images dataset [49] to demonstrate behavior on unseen objects and data distributions for methods trained on COCO 2017 dataset and classes. The results are shown in Fig. 5. All the methods demonstrate high object localization accuracy, where foreground region regression appears correlated with the choice of detection architecture. The KGE-based detection heads have consistently lower confidence on the detections, however, they skew the object classification to more semantically related classes, e.g. the unknown snow mobile in Fig. 5(b) is assigned to related classes such as car, snowboard, or motorcycle by the KGE methods, rather than airplane, boat, or truck by the baseline methods. The KGE methods also exhibit fewer false positives, such as the surfboard for Faster R-CNN in Fig. 5(a) or kite in Fig. 5(d) for DETR, which are not captured by the average precision metric on the object detection benchmarks.

Conclusions
In this work, we demonstrate the transfer of feature embeddings from natural language processing or knowledge graphs as class prototypes into two-stage, keypoint-based, and transformer-based object detection architectures trained using a contrastive loss. We performed extensive ablation studies that analyze the choice of class prototypes and distance metrics on the COCO 2017 and Cityscapes datasets. We showed that our resulting method can compete with their standard configurations on challenging object detection benchmarks, with error distributions that are more consistent with the object categories in the groundtruth. Especially, class prototypes derived from the ConceptNet knowledge graph [42] using a cosine distance metric demonstrate low inter-categorical confusions. Future work will investigate whether these knowledge embeddings in the classification head also benefit the class-incremental learning task in object detection. In this supplementary material, we provide additional insights and experimental results on knowledge embeddingbased object detection.

Object Type Representation
First, we inspect the knowledge embeddings that we incorporated as class prototypes into the object detection architectures. Fig. 1 summarizes the pairwise distances among a subset of the COCO 2017 [4] classes for each knowledge embedding. Please note that we exploit the symmetry of the distance metrics, and show cosine (upper triangle) and Manhattan distance (lower triangle) alongside in the same plot. The metrics share the main diagonal, as they are both zero for identical input vectors. The distances are bounded by d(x 1 , x 2 ) ∈ [0, 2], since the embedding vectors are normalized for the calculation of the cosine distance, respectively projected into the unit sphere for the calculation of the Manhattan distance.
In Fig. 1 (a), we extracted the class prototypes from the penultimate layer of a Faster R-CNN [1] model trained on the COCO 2017 dataset using a cross-entropy loss and learnable class prototypes. The method tends to maximize the pairwise distances among the class prototypes without semantic structure, except for weak clusters among groups of vehicle, animal, furniture, or food classes. Fig. 1 (b) -(d) show the fixed class prototypes that were used during the experiments. Compared to the learned prototypes in (a), the distances are less uniform, reflecting the inherent semantic structure of the embeddings. The COCO embeddings ( Fig. 1 (b)) have noticeably low interclass distances and depict clusters of commonly co-occurring objects such as vehicles, furniture, and pets, as well as foods.
In the GloVe embedding [16] (Fig. 1 (c)), we observe the most irregular distances end especially two outlier classes, such as bear or dining table, that have large distances to all other class embeddings. The ConceptNet embeddings [42] seem most similar to the learned embeddings with large inter-class distances, however, the inter-class distances show categorical differences, for example between furniture for sitting and lying, and vehicles for driving, swimming, or flying. The categorization of classes is shown in Fig. 2

Extended Ablation Study
In this section, we motivate the design choices we that make for the KGE object detection heads. We therefore ran experiments on the Faster R-CNN [1] base model using our proposed KGE extension as described in the main paper.

Explicit vs Implicit background representation
The application of nearest neighbor classification brings the design choice of how to represent the background class. Tab. 1 shows our results when comparing an explicit background representation by an additional background class prototype, or an implicit representation by a distance threshold to all foreground class representations. We found that an explicit representation results in marginally higher AP , however, this appears to be a misconception due to the nonweighted mean, while an implicit representation achieves higher AP w and AP category,w for the cosine distance metric. For Manhattan distances, the implicit representation surpasses the explicit representation on all metrics.
In our benchmark experiments, we choose an im- person  bicycle  car  motorcycle  airplane  bus  train  truck  boat  horse  sheep  cow  elephant  bear  cat  dog  chair  couch  toilet  bed  dining table  orange  banana  apple  carrot   person  bicycle  car  motorcycle  airplane  bus  train  truck  boat  horse  sheep  cow  elephant  bear  cat  dog  chair  couch  toilet  bed  dining table  orange person  bicycle  car  motorcycle  airplane  bus  train  truck  boat  horse  sheep  cow  elephant  bear  cat  dog  chair  couch  toilet  bed  dining table  orange  banana  apple  carrot   person  bicycle  car  motorcycle  airplane  bus  train  truck  boat  horse  sheep  cow  elephant  bear  cat  dog  chair  couch  toilet  bed  dining table  orange   plicit background representation, motivated by the larger AP category,w , which we interpret as an indicator of more semantically-grounded errors.

Contrastive vs. Margin Loss
We experimented with both contrastive and margin embedding losses. One representative of the latter is the Hinge embedding loss that, given a proposal embedding b i , minimizes the distance to the true class prototype while enforcing a fixed margin ∆ to all other class prototypes as follows: where y i,c is an indicator variable, that is 1 if a proposal box i is assigned a groundtruth bounding box of class c by a matching algorithm (e.g., IOU-based or Hungarian matching), and −1 for all other classes. In our experiments, we set the distance margin to ∆ = max{0.1, d(t c , t f )}, i.e., the distance between the true class embedding {t c |y i,c = 1} and all other class embeddings {t f |y i,f = −1} during training. Another hyperparameter is the negative sampling rate r which denotes the number of randomly selected class prototypes other than the groundtruth class embedding evaluated for one proposal box embedding, for which we compute the margin loss term. For these experiments, we sample r = 5 negative pairings uniformly from the set of COCO 2017 classes excluding the true class. As we can see from Tab. 2, the choice of embedding loss goes along with the distance metric that we use. A margin loss with a Manhattan distance metric achieves the highest AP and AP w . However, the contrastive loss performs better with a cosine distance metric and achieves the highest overall AP category,w with 43.5%. We choose the contrastive loss as it spares us two hyperparameters, the fixed margin ∆ and negative sampling rate r, with a negligible decline in average precision.

High vs. Low Temperature Contrastive Losses
In the next step, we evaluated the inverse magnitude scaling parameter in the contrastive loss function, the so-called    the interval d ∈ [0, 2], which results in flat scores after the approximate softmax scaling of the contrastive loss computation and thus low gradients. As we aim to achieve competitive performance as the baselines using cross-entropy losses, we follow [46] and fix τ = 0.07.

Hubness Reduction vs. No Hubness Reduction
Fei et al. [50] recently raised awareness for the hubness problem in image classification. The hubness problem is a phenomenon that concerns nearest neighbor classification in high dimensional spaces. It relates to the distance concentration effect and results in few class prototypes being particularly often among the nearest neighbors of embeddings, while others are almost never. They report that local feature scaling by z-score standardization can reduce hubness in image classification. In this work we aim to validate  [49] validation set using KGE configurations with cosine distance and ConceptNet [42] class prototypes. Please note that confidence scores for KGE methods describe the normalized similarity value between class prototype and visual embeddings.
whether z-score standardization can improve the classification accuracy as indirectly measured by the AP . The results presented in Tab. 4 shows that a hubness reduction measure such as z-score standardization does not improve AP for neither cosine nor Manhattan distance. On the contrary, it hinders convergence of the training, such that the resulting validation set performance is worse when using z-score normalization. Therefore, we conclude that the fixed knowledge embeddings already bring sufficient training data and internal structure to alleviate hubs in the feature space.

Extended Qualitative Results
In this section, we discuss the out-of-distribution predictions on the OpenImages [49] dataset shown in Fig. 3, for models trained on COCO 2017 data and classes. In general, we observe that the object detectors mostly detect and localize all foreground objects in the frame, even if they are from unknown classes. The KGE modification appears to predominantly impact the classification scores and results in more semantically-grounded class predictions. For instance, the velomobile in Fig. 3 (a) is classified as car or motorcycle by the KGE models, rather than boat, Frisbee, or airplane as observed in the predictions from the standard methods. The same holds for the pilot-shaped hot air balloon in Fig. 3 (b), which is consistently identified as a kite using the knowledge-embedded class prototypes.
The KGE methods also predict reasonable high confidences to more abstract objects such as the wheelchair or