{'fraction_non_alphanumeric': 0.039656060975940544, 'fraction_numerical': 0.015298234193180894, 'mean_word_length': 4.4058651026392965, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Incrementally Zero-Shot Detection by an Extreme Value Analyzer

Human beings not only have the ability to recognize novel unseen classes, but also can incrementally incorporate the new classes to existing knowledge preserved. However, zero-shot learning models assume that all seen classes should be known beforehand, while incremental learning models cannot recognize unseen classes. This paper introduces a novel and challenging task of Incrementally Zero-Shot Detection (IZSD), a practical strategy for both zero-shot learning and class-incremental learning in real-world object detection. An innovative end-to-end model -- IZSD-EVer was proposed to tackle this task that requires incrementally detecting new classes and detecting the classes that have never been seen. Specifically, we propose a novel extreme value analyzer to detect objects from old seen, new seen, and unseen classes, simultaneously. Additionally and technically, we propose two innovative losses, i.e., background-foreground mean squared error loss alleviating the extreme imbalance of the background and foreground of images, and projection distance loss aligning the visual space and semantic spaces of old seen classes. Experiments demonstrate the efficacy of our model in detecting objects from both the seen and unseen classes, outperforming the alternative models on Pascal VOC and MSCOCO datasets.


I. INTRODUCTION
This paper studies object detection, which, as one important computer vision task, has seen unprecedented advances in recent years with the development of Convolutional Neural Network (CNN). However, most successful object detection models, to data, are formulated as supervised learning problems in a batch setting. Such detection models have limited the capability of generalizing to unseen object classes, or being learned in an incremental setting. Humans, on the other hand, not only have the ability to recognize novel unseen object classes, but incrementally incorporate these novel object classes to existing knowledge preserved as well. For instance, a boy visiting the zoo will continuously identify and remember many new animals whilst not forget his pet at home.
Previous endeavors tackle object detection in either zeroshot learning (ZSL) [1], or class-incremental manner [2], [3]. Particularly, model learns to recognize and localize objects from unseen classes in Zero-Shot Detection (ZSD) [1], [4], Supported by NSFC Grant 71991471, and NSFS Grant 20ZR1403900. [5], [6], [7]. A successful ZSL model should have effective semantic knowledge transfer, i.e., transferring knowledge of seen classes, such as semantic attributes or word vector, to unseen classes. However, the ZSL assumes that all seen classes should be known beforehand, and the recognition model is trained in a batch; this greatly and unrealistically simplified the problem in the real-world setting. In contrast, the class-incremental learning (CIL) task [8], [9], [10] aims at continuously learning newly observed classes without forgetting existing old classes, i.e., catastrophic forgetting [11]. The CIL model is trained via a stream of data in which examples of different classes occur at different times. Notably, without semantic knowledge transfer as ZSL, CIL classifiers are not capable of inferring of objects of unseen classes in testing.
Critically, it would be desirable for vision systems to be able to perform incrementally zero-shot detection (IZSD). Particularly, referring to the properties of CIL in iCaRL [9], we quantify several properties of an algorithm qualified as IZSD: (1) an object detection model is trained from a stream of data, in which visual examples and semantic information of different object classes are observed at different incremental steps; (2) the model, at current incremental step, should not be updated by examples, from existing (old seen) object classes, but only from newly observed (new seen) object classes, which are unseen classes in previous incremental steps, as illustrated in Fig. 1. (3) The trained model should at any time provide a competitive detector for new seen, old seen and unseen object classes, (4) by the bounded, memory footprint, and computational cost, with respect to the number of seen object classes at each incremental step.
These criteria, essentially, identifies the key difference between IZSD and CIL/ZSL, as well as excluding the trivial solutions in memorizing all seen objects to retrain a zeroshot detector at each incremental step (the Fourth Criterion). Specifically, the IZSD should, in principle, maintain an updated incremental object detector by leveraging both visual and semantic representations of new seen object classes at each incremental step. In contrast, the naive solutions by either visual or semantic cues may be tempted to the deteriorated Fig. 1. Incrementally zero-shot detection. At each incremental step, examples of new seen object classes and semantic information are utilized to update the detection model. The model is able to detect objects from old seen, new seen, and unseen classes. If there is only one incremental step, then IZSD is equivalent to zero-shot detection. In addition, if semantic information is not utilized to detect unseen object classes, the model is equivalent to perform class-incremental detection. detection results. For example, one can directly incrementally update ZSD models by using the semantic information from new seen object classes. However, the predictions of such models may be biased towards the old seen object classes, as the corresponding visual feature extractor is not updated by the visual information from these new seen object classes. On the other hand, the typical class-incremental object detector [2], [3], updated by the visual cues (but not semantics) of new seen object classes, is not capable of localizing objects from those classes that are unseen at the current incremental step. These two naive baselines are compared in Tab. VI. A good IZSD model should be both efficient in semantic knowledge transfer, and robust to catastrophic forgetting. However, at one incremental step, the model may be prone to overfitting new seen classes, as they are blind to semantic embeddings of unseen object classes [12], and catastrophically forget the visual representations of old seen classes. Towards this technical difficulty of IZSD, we propose a novel extreme value analyzer (EVer), in differentiating the unseen, from both new and old seen classes by comparing the extreme values [13] of each class in the semantic spaces. Furthermore, derived from knowledge distillation, a novel old-new model is presented to help remember both the visual and semantic information of old seen classes.
Formally, we propose a novel end-to-end object detection model for IZSD by EVer (IZSD-EVer), with the key components of the old-new model and EVer. Specifically, with the backbone of Faster RCNN (FRCN) [14], the old-new model maintains the visual and semantic knowledge of old seen classes, by knowledge distillation [15] and projection distance loss, with a memory component introduced. Furthermore, inspired by Extreme Value Theory (EVT) [13], EVer, in the semantic space, measures the similarity of each instance to the mean vectors of instances from each seen class by generalized Pareto distribution (GPD) [16]. EVer, in the testing, infers the GPD probability of belonging to seen or unseen classes, one sample which is further predicted by either the incremental classifier or zero-shot classifiers. On the benchmark dataset, we extensively evaluate IZSD-EVer, and the results demonstrate the efficacy of IZSD-EVer in solving the IZSD task.
We make four main contributions in this paper: (1) We introduce a new object detection task of incremental zero-shot detection, a practical strategy for both ZSL and CIL in the real-world object detection. (2) We propose an innovative endto-end trainable model -IZSD-EVer in addressing IZSD by integrating ZSL and CIL in a single framework. (3) A novel extreme value analyzer, is presented, to simultaneously detect objects from old seen, new seen, and unseen classes. To the best of our knowledge, the GPD in EVT, is for the first time, introduced for both object detection and class-incremental learning. (4) Additionally and technically, we propose two novel losses, i.e., backgroud-forground mean squared error (bfMSE) alleviating the extreme imbalance of the background and foreground of images, and projection distance (PD) loss aligning the visual and semantic spaces of old seen classes.

II. RELATED WORKS
Zero-Shot Detection The task of ZSL is seeking to recognize unseen visual categories [17], [18], [19], [20], [21], [22]. As an extension, ZSD aims at recognizing and localizing the objects of unseen classes in testing [5], [1], [4], [23], [6], [7]. Specifically, they adapt the existing detectors (2-stage for [5], [1], [6], [7], 1-stage for [4], [23]) in the ZSD setting by adding a separate semantic prediction training task on the annotated seen objects, such that domain knowledge can be transferred to detect unseen objects which are semantically similar to seen objects. Among them, Rahman et al. [6] proposed polarity loss explicitly maximizing the margin between predictions for positive and negative classes based on focal loss. In contrast, our IZSD-EVer employs the proposed bgMSE loss as well as reconstruction loss and triplet loss. Generally, the ZSL assumes the model trained in a batch, whilst the proposed IZSD task requires model learning in a class-incremental manner, which is more realistic in real-world applications. Class-Incremental Detection Researchers have proposed many different approaches to solve class-incremental learning [10], [8], [9], [24], [25], [26]. However, these methods focus on solving image classification task rather than more difficult object detection task. Class-incremental detection aims at adding new classes to well-trained object detector incrementally, using only the data of the new classes, independent with totally re-training. Shmelkov et al. [2] presented the first approach to solve the task of class-incremental detection, by a two-stage object detection model on Fast R-CNN [27] with proposals generated by EdgeBoxes [28]. The model alleviates catastrophic forgetting by optimizing cross-entropy loss and knowledge distillation loss [15]. However, this model is not an end-to-end model and requires the external Edge-Boxes to generate proposals, which consume many computing resources. In contrast, Hao et al. [3] proposed an end-toend architecture for class incremental detection. Based on FRCN, they introduced knowledge distillation in both Region Proposal Network (RPN) and FRCN, transferring knowledge from teacher subnetwork to student subnetwork to generate high-quality box proposals, preventing catastrophic forgetting. Again, object detection in this setting, is different from our IZSD, as their incapable of predicting objects from unseen classes, and functionally, our EVer is proposed for such a purpose, inspired by recent works on open set recognition by extreme values in computer vision [29], [30], [31], [32], [33].
III. PROPOSED MODEL Problem Setup. In an incremental step, a set of old seen classes denoted as O, whose samples have been used to train the model in the previous incremental steps. Moreover, a set of new seen classes N , whose samples are training data of the current incremental step. There is also a set of unseen classes U, whose samples are only accessible during the test phase. These three sets of classes are mutually exclusive, i.e., O ∩ N ∩ U = ∅, so the set of all classes is denoted by C = O ∪ N ∪ U. The set of seen classes is S = O ∪ N . Besides, N O , N N , N U , N S , N C represent the number of old seen, new seen, unseen, all seen, and all classes, respectively. We denote semantic information as E ∈ R (C+1)×d composed of (C + 1) semantic embeddings (e.g., word2vec [34], GloVe [35], attributes) of d dimensions as its rows. Each object class has a corresponding semantic embedding, including the background class. The semantic embedding of the background class is the mean semantic embedding of other classes, i.e., E 0, , which is the same as [1]. Given a model well-trained on old classes dataset X O , the IZSD model is updated by new classes dataset X N at current incremental step, and is capable of generalizing to predict examples from O, N and U, individually. Additionally, we introduce a bounded memory M = {x i , i = 1, 2, . . . , K}, with memory size K is much smaller than the number of X O , i.e., K N O . End-to-End Learning of IZSD-Ever. The model is learned in an end-to-end way. In the first incremental step, since no existing old classes, the new model performs end-to-end learning on X N through the following loss function.
Then we estimate the parameters of GPD by maximum likelihood estimation (MLE) based on the projected semantic vectors for each new class. We store the most representative images for each new class in the memory component, as described in Sec. III-B. In the subsequent incremental steps, the new class dataset X N and memory M are simultaneously inputted into the old and new models, and the new model is trained end-to-end using the following loss function.
For the new classes, we estimate the parameters of the GPD for them as in the first incremental step. The parameters of GPD of the old classes remain unchanged. The memory M is also updated accordingly. We will introduce each loss in the following subsection.

A. Backbone with Semantic Embedding
Built upon the backbone of FRCN, we additionally add the semantic embedding. Particularly, as shown in Fig. 2, we denote the visual feature of the fully connected (FC) layer before zero-shot classifier as f (x; Θ) ∈ R v , where x and Θ denote the input image, the weights of FRCN model, individually. It is followed by another FC layer for zeroshot classifier, projecting visual features from visual space to d-dimensional semantic space denoted by W ∈ R d×v , as s = Wf (x; Θ), s ∈ R d . Cosine similarity is employed to measure the distance between projected semantic vector s and semantic embedding of classes E, and output the predicted class probability, as p zsc = softmax(Es), p zsc ∈ R N C +1 , where s and E are 2 normalized.
To efficiently learn the trainable W, we present three loss functions to help align visual features and with corresponding semantic embeddings. Particularly, bfMSE Loss. We train W using background-foreground MSE (bfMSE) loss to alleviate extreme imbalance between background and foreground in proposals generated by RPN. We separately compute MSE loss between the projected semantic vectors of background and foreground and the corresponding semantic embeddings. The bfMSE loss is defined as where I bg i and I obj i are indicators showing that the i-th region proposal contains background and object, respectively. N bg and N obj are the number of region proposals belonging to the background and foreground, respectively. y i is the ground-truth label of i-th region proposal. We empirically set α = 5. Reconstruction Loss. We introduce this loss in [36] to learn a more generalized model, as where W T project s back to visual space to reconstruct the visual feature. N p = N bg + N obj is the number of proposals. Then we compute the MSE loss between the reconstructed visual features and the original visual features. Note that L bf mse and L rec are to constrain the projected semantic vectors to be similar to semantic embedding. Triplet Loss. This loss enforces the projected semantic vectors are more similar semantic embedding of ground-truth classes, than those from other classes, defined as where C ij is the cosine similarity between the s i and E j,: . Similarly, C iyi is the cosine similarity between the s i and its ground-truth semantic embedding E yi,: . In addition, the standard losses in FRCN are also utilized here as L F RCN = L rpn cls + L rpn reg + L reg , where L cls rpn , L reg rpn and L reg are the classification loss in RPN, bounding box regression loss in RPN and bounding box regression loss of bounding box regressor, respectively.
The overall loss function of the backbone model is where β is a hyperparameter to control the importance of different losses.

B. The Old-New Model
The bounded memory component M [24], [26], [25], [9] is introduced into our old-new model to alleviate the imbalance between old seen and new seen classes. Particularly, M stores a few images for each seen classes based on their representative object example visual features f (x; Θ), which closest to the mean feature vector of the classes. Since M has a limited memory capacity of K, it will clear the least representative images of each class as new seen classes come.
The old-new model is presented to incrementally learn new seen classes without catastrophic forgetting old seen classes. Specifically, as in Fig. 2, we add an incremental classifier to the backbone model. When new seen classes N occurs at the current incremental step, we combine the old model learned in previous steps, with the new model learned in this step, forming the old-new model. When incremental learning is performed, new class dataset X N and memory M are simultaneously entered into the old-new model, where the parameters of the old model are fixed. When our old-new model learns new classes, the RPN of the new model needs to be able to generate region proposals containing objects of the old or new classes. We re-purpose the domain expansion in RPN of [3] to maintain the knowledge of the old class. Thus, we freeze the RPN classifier to keep the decision boundary of the RPN classifier unchanged and update the parameters of other parts of the RPN, making the feature distance loss.
f rpn (·) denotes the feature map generated by the intermediate convolutional layer of RPN. This loss ensures that the RPN generates proposals containing objects of new or old classes. When learning with new classes, the number of output nodes of the incremental classifier in the new model is expanded to the number of classes that have seen. As shown in Fig. 3, besides the traditional cross-entropy loss on all classes, we add a knowledge distillation loss [15] on the incremental classifier to transfer knowledge from the old model to the new model to alleviate performance dropping on old seen classes. Besides, we also add a projection distance (PD) loss on the zero-shot classifier to maintain the ability to align the visual space and semantic spaces of old seen classes. Particularly, let us denote the projected semantic embeddings of the old and new models as s o and s n , respectively. Similarly, we denote the output logits of the incremental classifier asŷ o andŷ n for the old and new models, respectively.
L cls = CE(y, softmax(ŷ n )), whereŷ o n is the output logits of the new model for old classes. y is the ground-truth label. T is a temperature scalar. CE(·) represents the cross-entropy loss. The overall loss function for incremental learning as follows: The first three items are multiplied by the ratio of the class they are responsible for, indicating their importance during training. It reduces the hyperparameters and makes the model easier to train. And γ is a hyperparameter.

C. Extreme Value Analyzer
We construct an EVer as the Pickands-Balkema-de Haan Theorem [37] to differentiate unseen from seen object classes. Typically, let X 1 , X 2 , . . . , X n be a sequence of identically distributed random variables (i.i.d) with cumulative distribution function F . The conditional excess distribution function of X over a threshold u is defined as (13) Theorem 3.1 (Pickands-Balkema-de Haan theorem): [37] The conditional excess distribution function F u can be well approximated by the generalized Pareto distribution when the threshold u is sufficiently large. That is The GPD is defined as x µ when ξ 0, µ x µ − 1 ξ when ξ < 0. In the semantic space, the projected semantic vectors of the same class are close together, and the projected semantic vectors of the different classes are far away from each other. As shown in Fig. 4, we propose to use the GPD of EVT to model the projected semantic vectors for each seen class. Particularly, if the distance between one projected semantic vectors to the mean projected semantic vector exceeds a threshold, such projected semantic vector would be taken as an extreme semantic vector. To this end, we fit the GPD based on the distance beyond the threshold.
We first compute the Euclidean distance d ij between the projected semantic vectors s i j and the mean projected semantic vector s j of j-th class, i.e., d ij = s i j −s j 2 , j ∈ {1, 2, . . . , S}, where s i j and s j are 2 normalized. Then we get the estimated parametersσ j andξ j of GPD by MLE based on the excess of distance exceeding threshold u j for j-th class. In classifying, we compute the excess of distance s−s j 2 beyond threshold Step Old classes New classes

Unseen classes
Train data Test data u j , and then derive the probability that the semantic vector is an extreme semantic vector by substituting the excess into the GPD, that is, the probability of not belonging to the j-th class. We get P min over seen classes as follows.
We obtain the prediction class label for the region proposal by the followinĝ y = arg max i∈{1,2,...,S} p ic i , P min < δ, arg max i∈{1,2,...,U } p zsc i , P min δ, where δ is a threshold. If P min δ, then s is an extreme semantic vector for all seen classes, the corresponding region proposal belongs to unseen classes. Conversely, if P min < δ, the corresponding region proposal belongs to seen classes.

A. Datasets and Settings
Dataset: We evaluate our model on Pascal VOC 2007 and 2012 datasets [38] and MSCOCO 2014 and 2017 datasets [39], respectively. MSCOCO is a large-scale dataset for object detection, segmentation, and captioning. This dataset is more challenging than Pascal VOC as it has 80 object classes, more small objects, and more complex background. Our model is compared in CIL and ZSL settings. Specifically, for classincremental detection, we use the same datasets VOC 2007 and MSCOCO 2017 as [3], [2]. For zero-shot detection, we use VOC 2007 and MSCOCO 2014 as [23], [6]. In order to compare fairly with other ZSD methods, we use the train split of VOC 2007 and 2012 (with 20 object classes) during training and use val+test split of VOC 2007 for testing. Evaluation Metric: We use the standard average precision (AP) at 0.5 IoU threshold over each class, and mean average precision (mAP) over all the classes as the evaluation metric. Classes Split: For class-incremental detection, we follow the classes split as [3]. As shown in Tab. X, we split all classes into four class groups denoted by G 1 , G 2 , G 3 and G 4 . Specifically, we sort all 20 classes of VOC 2007 alphabetically, split them into four class groups equally, so that each class group contains five classes. We remove images that contain objects from two or more class groups. As shown in Tab. X, we use D tr (·) and D te (·) to represent training and testing data that only contain objects of classes within the corresponding class group, respectively. For MSCOCO 2017, we use the same split method as for VOC 2007, except that the class order Step aero-plane  bicycle  bird  boat  bottle  bus  car  cat  chair  cow  dining  table  dog  horse  motrobike  person  potted  plant  sheep  sofa  train  tvmonitor  Seen  Unseen  mAP   1    in MSCOCO 2017 is specified by itself. For the seen/unseen classes split for ZSD, we follow the split for VOC 2007 and 2012 proposed by [23]. Particularly, car, dog, soft, and train as unseen classes, other classes as seen classes. There are two different types of seen and unseen classes splits setting for MSCOCO 2014: the 48/17 and 65/15 seen/unseen classes splits proposed by [5] and [6], respectively. Semantic Embedding: Many ZSD models use manual attributes and unsupervised word2vec/GloVe as semantic embeddings [1], [4], [40]. aPascal-aYahoo dataset [41] has 64dimensional binary attributes that characterize the visible objects for 20 object classes of Pascal VOC [40]. Therefore, we use the 64-dimensional attributes of aPascal-aYahoo dataset for Pascal VOC in our experiments. Inspired by [6], we use the l 2 normalized 300-dimensional unsupervised word2vec [34] trained on billions of words from texts like Wikipedia for MSCOCO in our experiments. Implementation: We use Faster RCNN as the backbone. As in Fig. 2, we use ResNet-101 as feature extractor that initialized by the weights pre-trained on ImageNet. Then we add one FC layer for Pascal VOC and three FC layers for MSCOCO followed by three branches predicting class probability for seen and unseen classes and bounding box coordinates. We empirically set α = 5 in Eq. 3, margin m = 1 in Eq. 5 and β = 0.001 in Eq. 7. T in Eq. III-B is set to 2 following the suggestion in [15]. In addition, we set γ = 2 in Eq. 12 like [3]. To set an appropriate threshold u j of GPD for j-th class, we sort d ij in the descending order, then set We choose it as 80% higher order quantile of the distances. We set δ = 0.02 in Eq. 17 by grid search. We set the memory size K = 150 and K = 600 for Pascal VOC and MSCOCO, respectively. Our model is trained by 10 epochs for each incremental step with batch sizes of 3 images per GPU. We set the learning rate by 0.001 at the beginning of each incremental step, and decrease it by 0.2 after 5 epochs.

B. Results on Incrementally Zero-Shot Detection
We rigorously evaluate our IZSD-Ever on both VOC 2007 and COCO 2017 datasets. We train our model by incrementally adding new classes and semantic information. As shown in Tab. X, in the first incremental step, we train the model on D tr (G 1 ) containing class group G 1 , and in the subsequent incremental steps, new classes are added continuously and trained using the corresponding train data. Tab. II and Tab. III present the results on IZSD of our IZSD-Ever. As shown in Tab. II, our model not only can keep the knowledge of the old class well, but also can effectively detect unseen classes. However, the performance of unseen classes is lower than that of seen classes, which is caused by the inherent domain drift problem [40] of ZSL. It refers to the difference in the data distribution of seen classes and unseen classes, which leads to performance degradation in unseen classes. We have also observed that some unseen classes perform poorly. This is because the unseen class needs to be semantically similar to some seen classes to effectively transfer knowledge from the seen classes to the unseen classes recognition task. Therefore, a good seen/unseen classes split can improve performance on unseen classes, as described in seen/unseen classes split in classes split IV-A. In addition, in Tab. III, the performance of our model in COCO2017 is relatively worse than that in VOC2007. On the one hand, it is because COCO is more challenging than VOC2007. On the other hand, compared with seen/unseen classes split of MSCOCO, too many unseen classes result in poor performance on unseen classes.

(3) CIFRCN NP [3] (a variant of CIFRCN). (4) Faster RCNN (fine-tuning)
is trained only on data in new classes but no old classes. (5) Faster RCNN (retrain) is trained on both new classes data and old classes data, whose performance can be seen as an upper bound of our framework can achieve. Results: We evaluate our IZSD-Ever and compared methods under the class-incremental detection setting. Tab. III presents the results on class-incremental detection of our IZSD-Ever and compared methods on VOC 2007 and COCO 2017. Our model is better than the compared methods in most cases, not only can keep the knowledge of the old class well, but also can effectively detect unseen classes. However, the compared methods cannot detect unseen classes. In the fourth step, our model achieves high performance in four class groups for the reason that all classes are seen classes at this step. This also shows that the performance of the previous three steps of seen classes is not very high because the need to detect seen and unseen simultaneously. Therefore, IZSD is a very challenging task, and it is still far from totally addressing this task.

D. Results on Zero-Shot Detection
Compared Methods: Our model compares to two recent ZSD methods on zero-shot detection. (1) HRM [23] proposes a convex combination of embeddings used in conjunction with a Step Baseline-L bone Baseline-SI G1 G2 G3 G4 mAP G1 G2 G3 G4 mAP 1 24. 17  (2) PLZSD [6] proposes a 'Polarity loss' to maximize the gap between positive and negative predictions. Results: We validate the performance of our model on ZSD tasks using VOC2007 +2012 and COCO 2014 datasets. As shown in Tab. IV, we present the results of our model and PLZSD on two different seen/unseen classes splits, and report mAP on the unseen classes for the zero-shot detection task.

E. Ablation Study
Compared with Two Baselines: We compare our model with two baselines under the setting of incremental learning on VOC 2007. Baseline-L bone : the model is trained using L (Eq. 1) along. Baseline-SI: the model that only adds the semantic information of the new(unseen) classes at from the second incremental step based on the data of the classes group G 1 , to recognize more and more new classes. Tab. VI demonstrates that the performance of the Baseline-L bone model in the old classes is significantly reduced, that is, catastrophic forgetting. The comparison between Tab. VI and Tab. III shows that L IL can effectively alleviate catastrophic forgetting. As shown in Tab. VI, even though the Baseline-SI method can alleviate forgetting on the class group G 1 , there is a highly biased toward predicting the seen classes for the reason that the model is trained using data from the seen classes. Compared with Tab. III, the performance of our model on VOC 2007 on class group G 1 is significantly better than that of Baseline-SI on class group G 1 . At the same time, the performance of our model in the unseen class is also better than that of Baseline-SI. Although Baseline-SI seems to be able to perform incremental learning using only the semantic information of the new classes, baseline-SI performs very poorly when there are more and more new classes. The Effect of bfMSE: To demonstrate the effect of bfMSE loss (Eq. 3), we compare the results of the model using general MSE loss on VOC 2007. As shown in the Tab. VII, the performance of our bfMSE on both seen and unseen classes is better than general MSE loss. This shows that using our proposed bfMSE loss can effectively reduce the impact of this background and foreground imbalance. More results are available in the supplementary material.

V. CONCLUSION
In this paper, we propose an end-to-end model IZSD-Ever to solve IZSD by integrating ZSL and CIL in a single model. We also propose a novel extreme value analyzer to detect objects from old seen, new seen, and unseen classes simultaneously. Besides, we designed two novel losses, i.e., bfMSE loss and projection distance loss for zero-shot classifier. In our experiments, our model outperforms the alternative methods on Pascal VOC and MSCOCO datasets.

SUPPLEMENTARY MATERIAL
In this supplementary material, we provide more implementation details and experimental results about our proposed framework. We first describe the details of the four datasets used in our all experiments. Then we provide details of classes splits of different datasets in incremental learning and zeroshot detection. We also provide some qualitative results of our IZSD-EVer on VOC 2007. Finally, we provide more results.

A. Datasets
In this section, we describe the details of the four datasets used in our experiments. As shown in Tab

B. Classes splits
In this section, we detail the classes splits of different datasets in incremental learning and zero-shot detection. In order to compare with two state-of-the-art class incremental detectors IDWCF [2] and CIFRCN [3], we use the same settings as [3]. As shown in Tab X. We sorted the 20 classes of VOC 2007 alphabetically and then divided them into 4 class groups equally. As for COCO 2017, we divide all classes into 4 class groups according to the class order of the dataset itself. For the seen and unseen classes split of zero-shot detection, we follow the classes split method for VOC 2007 + VOC 2012 proposed in [23]. We also follow two different types of seen and unseen classes splits setting for MSCOCO 2014: the 48/17 and 65/15 seen and unseen splits proposed by [5] and [6], respectively.

C. Implementation Details
In this section, we will describe more implementation details. We use Faster RCNN as the backbone. We follow [42] that use those layers before conv5 x of ResNet-101 as feature extractor, while conv5 x is performed on RoI pooled feature map. The threshold δ used to determine whether the region proposal belongs to unseen is generally very small, and we set it to 0.02 in our experiments. We set the memory size K =

D. Qualitative Results
As shown in Fig. 6, we provide some qualitative results of our IZSD-EVer on VOC 2007.

E. The fitting of GPD
In EVer, we use the GPD to model the Euclidean distances from projected semantic vectors to the mean semantic vector for each seen classes. As shown in the left part of Fig. 5, the histogram of the Euclidean distances presents a thick tail distribution, so EVT is well suited for modelling the Euclidean distances. To test the fitting of GPD, we used the most commonly used Quantile-Quantile (Q-Q) plot. Q-Q plot is a graphical technique for determining whether a certain two data sets are from the same distribution.
As can be seen from the right part of Fig. 5, Q-Q plot is very approximate to a straight line, indicating that GPD fitting is very well.

F. The effect of PD loss
To demonstrate the effect of PD loss, we compare the results of the model without PD loss. As shown in the Tab. IX, the performance of our model on all classes is better than the model without PD loss. This shows that using our proposed PD loss can maintain the ability to align the visual space and semantic spaces of old seen classes.

G. The effect of α in L mse
The hyperparameter α in L mse is used to alleviate extreme imbalance between background and foreground in region proposals generated by RPN. To demonstrate the impact of α, we fixed other hyperparameters to evaluate the performance of our IZSD-EVer with different α = {1, 3, 5, 7}. As shown in Fig. 7(a), our IZSD-EVer with α = 5 performs the best for seen and unseen classes. Therefore, we set α = 5 in the main paper.
H. The effect of β in L zsc β is a hyperparameter to control the importance of L rec loss. We also do comparative experiments to evaluate the effect of β in L zsc . We fixed other hyperparameters and only changed the β to {0.1, 0.01, 0.001, 0.0001}. As shown in Fig. 7(b), our IZSD-EVer performs the best for seen and unseen classes where β = 0.001. Therefore, we set β = 0.001 in the main paper.

I. The effect of γ in L IL
The hyperparameter γ controls the importance of L rpn f d in L IL . We do comparative experiments to evaluate the effect of γ. We fixed other hyperparameters and only changed the γ to {1, 2, 5, 10}. As shown in Fig. 7(c), our IZSD-EVer performs the best for seen and unseen classes where γ = 2, which is consistent with the recommendation of Hao et al.. Therefore, we set γ = 2 in the main paper.

J. The effect of margin m in L tri
The hyperparameter m is the margin in the L tri . To demonstrate the impact of margin m, we fixed other hyperparameters and only changed the margin m to {1, 2, 5, 10}. As shown in the Fig. 7(d), as m increases, the mAP of IZSD-EVer shows a slow downward trend. Our IZSD-EVer performs the best where m = 1. Thus, we set m = 1 in the main paper.
K. The effect of δ in EVer δ is the threshold used in EVer to distinguish whether the projected semantic vector comes from the seen classes or the unseen classes. To demonstrate the impact of δ, we fixed other hyperparameters and only changed the δ to {0.01, 0.02, 0.05, 0.1, 0.2}. As shown in the Fig. 7(e), our IZSD-EVer performs the best where δ = 0.02, so we set δ = 0.02 in the main paper.
L. The effect of memory size K The bounded memory component M is introduced to alleviate the imbalance between old seen and new seen classes. To demonstrate the impact of memory size K, we fixed other hyperparameters to evaluate the performance of our IZSD-EVer with different K = {150, 300, 450, 600}. As shown in the Fig. 7(f), as K increases, the mAP of IZSD-EVer gradually increases. However, the memory size cannot be too large, especially for devices with limited storage capacity, so we set K = 150 in the main paper.