{'fraction_non_alphanumeric': 0.03144943368021618, 'fraction_numerical': 0.011815098027942275, 'mean_word_length': 4.84066487575554, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Few-Shot Object Detection by Knowledge Distillation Using Bag-of-Visual-Words Representations

While fine-tuning based methods for few-shot object detection have achieved remarkable progress, a crucial challenge that has not been addressed well is the potential class-specific overfitting on base classes and sample-specific overfitting on novel classes. In this work we design a novel knowledge distillation framework to guide the learning of the object detector and thereby restrain the overfitting in both the pre-training stage on base classes and fine-tuning stage on novel classes. To be specific, we first present a novel Position-Aware Bag-of-Visual-Words model for learning a representative bag of visual words (BoVW) from a limited size of image set, which is used to encode general images based on the similarities between the learned visual words and an image. Then we perform knowledge distillation based on the fact that an image should have consistent BoVW representations in two different feature spaces. To this end, we pre-learn a feature space independently from the object detection, and encode images using BoVW in this space. The obtained BoVW representation for an image can be considered as distilled knowledge to guide the learning of object detector: the extracted features by the object detector for the same image are expected to derive the consistent BoVW representations with the distilled knowledge. Extensive experiments validate the effectiveness of our method and demonstrate the superiority over other state-of-the-art methods.

. We learn a representative bag of visual words (BoVW ) using the proposed PA-BoVW model. For an extracted positive proposal, we encode it with BoVW in a pre-learned feature space and the feature space of the object detector, respectively. Then we perform knowledge distillation by matching two BoVW representations to guide the learning of the object detector.
classes with only a few samples available. Thus, few-shot object detection eliminates exhaustive label annotation of massive data on novel classes. Compared to general object detection [27,31,32,40], few-shot object detection [3,19] is much more challenging due to the difficulty of learning generalizable features that can be transferred from base classes to novel classes.
A classical type of methods for few-shot object detection is fine-tuning based methods [1,9,26,30,38,44,48,49,57], which first train the object detector using the samples from base classes, then fine-tune the model on novel classes. A prominent example is TFA [44], which first adopts such two-stage training strategy to transfer knowledge from base classes to novel classes. Based on such fine-tuning framework, many methods are proposed to deal with various challenges of fewshot object detection. Typical methods include FSCE [38] aiming to facilitate the separability among similar classes, MPSR [49] which seeks to rectify the sample distribution for novel classes, and HallucFsDet [57] which is designed to tackle the problem of data scarcity.
A crucial challenge of fine-tuning based framework for few-shot object detection is the potential class-specific overfitting on base classes and sample-specific overfitting on novel classes. On the one hand, although sufficient samples are provided for base classes, the object detector is still prone to overfitting on base classes during the first stage of training process. In this case, the detector learns the class-specific features instead of the class-agnostic features, which cannot be transferred to novel classes and would adversely affect object detection for novel classes. On the other hand, owing to the scarcity of training samples for novel classes in the fine-tuning stage, the object detector tends to be overfitting on these individual samples and thus learns sample-specific features that cannot be generalized across different samples for a same novel class.
To address above limitation, we propose to perform knowledge distillation to guide the learning process of few-shot object detection and thus restrain the potential overfitting on both base classes and novel classes. As shown in Figure 1, we propose the novel Position-Aware Bag-of-Visual-Words (PA-BoVW ) model, which is able to learn a bag of visual words (BoVW ) from a limited size of image set. The learned visual words are representative and comprehensive to be capable of encoding general images based on the similarities between the learned visual words and an image. Then we can perform knowledge distillation based on the intuition that an image should have consistent BoVW representations in two different feature spaces, provided that the image is encoded properly, namely not overfitted, in both feature spaces. Concretely, we first pre-learn a feature space and derive a BoVW representation for an image in this space. The obtained BoVW representation can be considered as distilled knowledge to guide the learning of object detector: the extracted features by the object detector for the same image are expected to derive the consistent BoVW representation with the distilled knowledge.
Unlike typical way that identifies visual words as the clustering centroids in the deep feature space [11,18], we learn visual words as learnable vectorial embeddings. To be specific, our proposed PA-BoVW model first constructs an effective deep embedding space for learning the visual words by training a backbone network in a self-supervised way. Then the visual words is learned in this embedding space in a supervised way employing image classification as a pretext task. Besides, we employ DeCov loss [5] as an auxiliary loss to reduce the inter-word redundancy and encourage the diversity of visual words. As a result, the PA-BoVW model is learned in an independent way from the task of object detection. Thus the encoded BoVW representation in its embedding space can be used as distilled knowledge, which can be transferred to the learning process of the detector to restrain potential overfitting on both base and novel classes.
To conclude, we make following contributions: 1) We propose the novel PA-BoVW model, which constructs an effective embedding space to learn a representative vocabulary of visual words; 2) Based on the PA-BoVW model, we design a knowledge distillation framework to guide the learning of the object detector and thereby restrain the potential overfitting on both base classes and novel classes; 3) Extensive experiments validate the effectiveness of our method and demonstrate the advantages of our method over state-of-the-art methods for few-shot object detection.

Related Work
Few-shot learning. Early works of few-shot learning focus on the task of image classification. Metric-based methods learn a suitable embedding space, where samples can be categorized correctly via a nearest neighbor classifier with Euclidean distance [37], cosine similarity [4,41] or graph distance [21,36,53]. Initialization-based methods aim to learn good initialization so that the model can adapt to novel tasks by a few optimization steps [10,23]. Hallucination-based methods alleviate data scarcity issue via learning generators to augment novel classes [14,45]. However, these approaches could not be directly applied to fewshot object detection which requires both classification and localization.
Few-shot object detection. Few-shot object detection aims to detect objects with few annotated training examples provided. There are several early methods adopting the idea of meta-learning [8,13,17,19,24,50,52,56]. FSRW [19] is a novel few-shot detector based on YOLOv2 [31], which re-weights the features with channel-wise attention and leverages these features to detect novel objects. Meta R-CNN [52] applies similar feature re-weighting scheme to Faster R-CNN [32] and performs meta-learning over RoI features. These methods usually suffer from a complicated training process and fail to learn generalizable features that can be transferred from base classes to novel classes. Recently, several fine-tuning based methods [1,9,26,30,38,44,48,49,57] achieve higher performance compared to meta-learning based methods. TFA [44] performs a simple two-stage finetuning approach which fine-tunes only the last layer on novel classes. MPSR [49] proposes to generate multi-scale positive samples to solve the problem of scale variations. FSCE [38] provides a strong baseline which fine-tunes feature extractors during the fine-tuning stage and employs a contrastive branch to rescue misclassifications. However, all these fine-tuning based methods suffer from overfitting on both base classes and novel classes. In this work, we design a novel knowledge distillation framework to tackle the problem. Bag of visual words. Bag-of-Visual-Words is a popular technique for image recognition. Many variants of BoVW have been proposed in the past [6,22,42] and they continue to be widely used in recent deep learning approaches [11,12,18,34]. VWE [34] designs a visual words learning module to generate CAMs [58] for weakly-supervised semantic segmentation. BoWNet [11] and OBOW [12] apply BoVW to self-supervised learning. QuEST [18] introduces to distill the quantized feature maps from the teacher to the student. In this work, we learn a BoVW model via a self-supervised task and image classification task. Knowledge distillation for object detection. Knowledge distillation [16] is an effective way to transfer knowledge acquired in teacher network to student network. Early works focus on the task of image classification [33,39,55]. Recently, there are several works which propose to transfer knowledge for object detection. Chen et al. [2] distill knowledge from the teacher detector to the student detector in all components (i.e., feature extraction, RPN, classification and regression networks). Wang et al. [43] design a fine-grained feature imitation method which distills the features from foreground area. In this work, we propose a novel knowledge distillation method which transfers knowledge from a BoVW model to a few-shot object detector, aiming to suppress potential overfitting on both base and novel classes.

Method
To deal with the potential overfitting in few-shot object detection based on deep learning networks, we propose to perform knowledge distillation to guide the learning process of few-shot object detection. Specifically, we learn a bag of visual words (BoVW ) for encoding images. The learned visual words are presumably representative, hence an image should have consistent BoVW representations in two independent feature spaces. We first pre-learn a feature space and the derived BoVW representation in this space can be considered as distilled knowledge that is transferred to the learning of the few-shot object detector. The extracted features by the object detector for each positive proposal are expected to derive consistent BoVW representation with the distilled knowledge, thereby avoiding overfitting during supervised learning.
In this section, we will first elaborate on the proposed Position-Aware Bag-of-Visual-Words model (PA-BoVW ) for learning a bag of visual words and encoding images based on these visual words. Then we will show how to perform knowledge distillation to guide the learning of few-shot object detection.

Position-Aware Bag-of-Visual-Words Model
A typical way of constructing a bag of visual words from an image corpus is to cluster the image patches in the corpus in deep feature space and select the clustering centroids as the visual words [11,18]. While such an unsupervised method is straightforward and feasible given a sufficiently large image corpus, it shows limited effectiveness when the size of image corpus is limited. This is mainly because such a method tends to focus on the statistically frequent image patches which may not be semantically representative.
In this work we present a novel Position-Aware Bag-of-visual-Words (PA-BoVW ) model, which is able to learn a representative vocabulary of visual words from a limited size of image set. As shown in Figure 2 (a), we view each visual word as a learnable embedding and learn the parameters of all word embeddings in a supervised way based on two pretext tasks. The first pretext task is Pixel-to-Propagation Consistency [51], which trains the backbone network to construct the embedding space for learning visual words in a self-supervised learning framework. Then we perform image classification on the representations encoded by our PA-BoVW model. Hence, the task of image classification serves as the second pretext task for optimizing the parameter learning of both the word embeddings and the backbone network. To this end, we first construct an iconic-object image dataset D as image corpus by extracting all base class objects from the detection dataset according to their ground-truth bounding boxes and labels. We then train our PA-BoVW model on D. Learnable word embeddings. Unlike the typical way that identifies visual words as the clustering centroids in the deep feature space, we learn visual words as learnable vectorial embeddings. A prominent benefit of such way is that the visual words do not necessarily correspond to image patches in the corpus. Instead, our model can learn an effective vocabulary of word embeddings freely from the whole embedding space under the optimization of the designed supervision (i.e., the classification pretext-task in our case). Self-supervised learning of embedding space via Pixel-to-Propagation Consistency. To construct an effective embedding space for learning visual words, we employ a backbone network to encode the input image into a latent feature space and optimize the backbone network using Pixel-to-Propagation Consistency (PPC) [51] in a self-supervised way. PPC optimizes the backbone to make each pixel distinguishable from other pixels in the embedding space.
Formally, given an image corpus D, for an image I ∈ D, two views (I 1 , I 2 ) are generated by typical data augmentations (random cropping, color distortion, etc.). They are then fed into a self-supervised framework including a regular encoding network and a paired momentum encoding network to extract features respectively, as shown in Figure 2 (a). The encoding network consists of a backbone network and a projection network. The features which pass the backbone networks are denoted as (F 1 , F 2 ). Then the projection networks convert them to (E 1 , E 2 ). Each pixel in the feature maps that pass the regular encoding network are processed by a pixel propagation module [51] to enrich its feature by attending to all other pixels in the same view according to their similarities, for instance, the pixel x 1 i in E 1 is processed as: Ψ (·) is a feature transformation module comprising 2 convolution layers with a batch normalization layer and a ReLU layer. The obtained feature q(x 1 i ) is then used to maximize its cosine similarities with its corresponding pixel x 2 i in the other view E 2 passing the momentum backbone: Note that x 1 i and x 2 i are the corresponding pixels in two views (E 1 , E 2 ), but they could have different positional coordinates. Since only the feature maps going through the regular backbone are processed by PPC [51], both views have chance to go through the regular backbone and the momentum backbone respectively.
Since the embedding space is constructed independently from the learning of the object detector, the embedding space and the feature space of the object detector are two different feature spaces. Thus, the constructed embedding space can be used to not only learn the vocabulary of visual words, but encode BoVW representations based on the learned visual words for an image as distilled knowledge. Such distilled knowledge is further used to guide the learning of the object detector. Position-aware encoding of BoVW representation. Typical way of representing images using the visual words is to divide an image into patches and calculate the histogram over the visual words [22]. However, such encoding scheme loses the position information which is crucial for object detection. To address this limitation, we encode an image using the vocabulary of visual words by calculating the Cosine similarity between each pixel of the image and each visual word in the embedding space while retaining the positional relationship among pixels. Formally, given an image I, its features which pass the backbone network are denoted as F ∈ R C×H×W with C feature maps of size H × W . The learned vocabulary of visual words are denoted as V ∈ R K×D in which K is the number of visual words and D is the feature dimension for each word. The Cosine similarity between the pixel (h, w) in F and the j-th word in V is calculated as: where F conv is a transformation function implemented by a convolutional layer to project F from C channels to D channels. Consequently, we obtain a similarity map P ∈ R K×H×W as the encoded BoVW representation for the image I, which retains the position information for each pixel. Supervised learning of visual words by the pretext task of image classification. We employ image classification as a pretext task to guide the learning of the visual words based on two considerations: 1) the visual words should be discriminative for object recognition in that our goal is object detection; 2) the learned visual words should have well generalizability and can be used for knowledge distillation to restrain potential overfitting for object detection, thus the pretext task should be independent of the task of object detection. Given an image I ∈ D, we first encode it with BoVW. Then the obtained BoVW representation is fed into a simple classification head consisting of a pooling layer and a fully-connected layer. Formally, we perform average pooling over the BoVW representation of I, namely the similarity map P, along the H and W dimension and thus achieve a vectorial representation whose dimension is equal to the size of the vocabulary V: The obtained P avg are further fed into a fully-connected layer F fc and Softmax function F softmax for classification prediction. Cross Entropy (CE) loss is used for optimization: L cls = CE(y I , F softmax (F fc (P avg ))), where y I is the groundtruth label for image I.
Reducing inter-word correlation. To encourage the diversity of visual words and reduce the redundancy among words, we add DeCov loss [5] as an auxiliary loss to minimize the correlation between different visual words: where Σ(·) denotes the covariance matrix and diag(·) extracts the diagonal elements of a matrix.
The whole Bag-of-Visual-Words model can be trained under the supervision by above three loss functions jointly: Note that although our PA-BoVW model is learned independently from the object detector in an extra step, it can be trained quite efficiently due to small size of object images and relatively simple supervision tasks compared to the task of object detection.

Knowledge Distillation for Object Detection
Our Position-Aware Bag-of-Visual-Words (PA-BoVW ) model learns the embedding space for visual words using PPC [51] as the pretext task, and learns the vocabulary of visual words based on the pretext task of image classification. Thus, our PA-BoVW model is optimized in a completely independent way from the task of object detection. As a result, the encoded BoVW representation in the embedding space of the PA-BoVW model for an image can be viewed as distilled knowledge, which can be transferred to the learning process of few-shot object detection to suppress potential overfitting on this image. The rationale behind this design is that a well learned (non-overfitting) feature representation for an object by a detector should bear consistent similarity distribution over the learned visual words with the corresponding BoVW representation by our PA-BoVW model. Thus, we can derive consistent BoVW representations from the learned features by the object detector and our PA-BoVW model, respectively. As shown in Figure 2 (b), we adopt the typical object detection framework, which is built upon Faster R-CNN [32]. Actually, our proposed method of knowledge distillation can be readily integrated into any classical object detection framework. Given a positive region proposal r generated by RPN (Region Proposal Network), which is assigned with one of the ground-truth labels and bounding boxes during training, we crop the corresponding region from the original input image and resize it to a fixed size by bilinear interpolation, then fed it into our PA-BoVW model to obtain its BoVW representation P(r) ∈ R K×H×W by Eq. 3. Meanwhile, we calculate the Cosine similarities between the features G(r) for r, obtained from the RoI pooling layer of the object detector, and the vocabulary of visual words V. Note that G(r) and the visual words are not learned in the same feature space, thus we project them into the same feature space first and then compute the Cosine similarities in the same way as Eq.3: Herein, g(·) is the project function implemented as a fully connected layer for V, while ϕ(·) denotes the project function for features G(r), which is formulated as a 1 × 1 convolutional layer. K, H, W are the size of the vocabulary V, the height and the width of G(r), respectively. The obtained Q(r) ∈ R K×H×W is equivalent to the BoVW representation encoded on the learned features G of the object detector. If G(r) is learned well and not overfitting on the input data, it should result in consistent BoVW representation as our pre-trained PA-BoVW model. Thus, we minimize the L1norm distance between these two BoVW representations to guide the learning process of the object detector: where R is the number of positive proposals. Knowledge distillation on both base classes and novel classes. As most fine-tuning based methods [30,38,44,49] do, we first train the object detector on base classes and then fine-tune the model on the novel classes. We perform the knowledge distillation process in both training stages to restrain the potential overfitting on base classes and novel classes, respectively. Collaborative object detection with BoVW representations. The obtained BoVW representation Q(r) can also be used for object classification for the region proposal r. Thus, we perform classification by fusing the predicted scores from Q(r) and the original features respectively: where p orig and p ′ are predicted scores from the original features and from the BoVW representation respectively. Here p ′ is obtained by performing a linear transformation F fc and softmax function on Q(r). η is a hyper-parameter to fuse two scores. During training, Cross Entropy loss is used as an auxiliary loss to guide the optimization: where y is the groundtruth label for the region proposal r.
Consequently, the object detector is trained under the supervision of three losses jointly: where L det corresponds to the standard Faster R-CNN [32] losses for object detection, including the losses for RPN, classification and box regression.

Experimental Setup
Benchmarks. We evaluate our approach on PASCAL VOC [7] dataset and MS COCO [28] dataset. We follow the previous work [19] for data construction to have a fair comparison. PASCAL VOC comprises 15 base classes and 5 novel classes. We utilize the same three class splits introduced in [19], where each novel class has k = 1, 2, 3, 5, 10 instances sampled from the combination of VOC 2007 and VOC 2012 trainval sets. VOC 2007 test set is used for evaluation. As for MS COCO, the 60 categories disjoint with PASCAL VOC are selected as base classes, and the remaining 20 categories are used as novel classes with K = 10, 30. For evaluation metrics, we report AP50 of novel classes (nAP50) for PASCAL VOC and COCO-style AP of the novel classes for MS COCO. Implementation Details. We evaluate our approach by building it upon two state-of-the-art methods: TFA++ [38] and DeFRCN [30]. TFA++ [38] is a strong baseline which jointly fine-tunes the feature extractors and box predictors during the fine-tuning stage. DeFRCN [30] is a simple yet effective architecture which is the current state of the art.
For PA-BoVW model, we use an ImageNet [35] pre-trained ResNet101 [15] as the backbone. The input size is 224 × 224. The feature dimension of visual word is 512. The number of visual words is set to 256 for PASCAL VOC and 1024 for MS COCO. We follow the same data augmentation strategy in PPC [51], where two random patches of an image are independently sampled, followed by random horizontal flip, color distortion, gaussian blur, and solarization. We use AdamW optimizer to optimize the PA-BoVW model with the initial learning rate of 1e-4 for 24 epochs. We decay the learning rate by ratio 0.1 at epoch 18 and 22. The total batch size is set to 256. The object detector is trained on 8 GPUs with a batch size of 16. The η is uniformly set to 0.5. All other training settings are the same as that in TFA++ [38] and DeFRCN [30].

Comparison with State-of-the-art Methods
Results on PASCAL VOC. Table 1 presents the results on PASCAL VOC, which show that our approach improves the performance of TFA++ [38] by a large margin in all cases including different numbers of training shots in different splits. Particularly, for the 2-shot case of Novel Split 1, 5-shot case of Novel Split 2 and 2-shot case of Novel Split 3, our approach is 8.1%, 6.5%, 5.5% higher than the baseline. When applying our approach to DeFRCN [30], which is the Table 1. Comparison with existing few-shot object detection methods using nAP50 as evaluation metric on three PASCAL VOC Novel Split sets. 'Ours (KD-TFA++)' denotes our method using TFA++ [38] as the baseline. † indicates that model is evaluated using the released code. current state of the art, our method still improves the performance in most cases, especially in the extremely-few-shot regimes such as 1-shot and 2-shot. Results on MS COCO. Table 2 shows the results on MS COCO. Applying our approach to two baselines achieves 1.1% and 0.3% nAP performance gain for 10-shot, 0.6% and 0.1% in terms of novel AP performance gain for 30-shot, respectively. There is no as large performance gain as on PASCAL VOC, which is probably because MS COCO has much more training images and thus has a lower risk of overfitting. To validate this speculation, we further evaluate our method by only using a small subset of base-class data for training. Specifically, we randomly select 10% samples from base classes to form a training set. For novel classes, we keep the same setting in standard few-shot object detection. Table 3 shows that the performance gains are larger than using all training data.

Ablation Studies
In this section, we conduct ablation studies on the Novel Split 1 of PASCAL VOC using TFA++ [38] as the baseline. Effect of each functional component. Table 5 shows the efficacy of each functional components for few-shot object detection on novel classes, including distillation on base classes, novel classes and score fusion for classification in Equation 10. With the knowledge distillation for base classes, the performance gain is 4.2%/2.5%/1.9% for 3/5/10-shot, respectively. By performing distillation on novel class during the fine-tuning stage, the performance gain increases  0.1%/0.9%/0.4% respectively, which indicates the improved generalization ability on novel classes by our method. Finally, fusing the predicted scores from the original features and from the BoVW representation for classification yields the extra performance gain by 0.9%/1.5%/1.1%. Effect of each loss function. Table 4 shows the effect of each loss function. Both the distillation loss L distill and collaborative detection loss L cls-BoVW improve the performance distinctly. Comaring the results in the last two rows, the L DeCov which is designed to encourage the diversity of visual words and reduce the redundancy among words, also improves the performance. Quantification for the overfitting on base classes. Fine-tuning training strategy tends to make models overfit on base classes. Since most parameters of the feature extractors are freezed or just fine-tuned slightly during the fine-tuning stage, most model capacity is allocated to fitting the base samples. To quantify such overfitting on base classes, we perform three experiments: 1) Using finetuning training strategy, we first pre-train the baseline TFA++ on base classes, then fine-tune it with sufficient novel-class samples instead of k shot per class, which is denoted as two-stage training mode; 2) Similar to the setting in 1), we fine-tune our model with sufficient novel-class samples after pre-training; 3) we train the baseline using sufficient samples for both base and novel classes together in one-stage mode. We compare both nAP50 and the number of misclassified samples from novel to base classes to measure the overfitting. The results in Table 6 show that 1) the baseline trained in two-stage mode performs much worse than training itself in one-stage mode and has a larger number of misclassified samples, indicating that the model is heavily biased towards base classes; 2) using the same two-stage training mode, our method achieves 3.5% of performance gain than baseline and substantially decreases misclassified cases, which demonstrates the effectiveness of our method for suppressing the overfitting on base classes.  Effect of different pretext tasks for learning BoVW models. We conduct experiments to compare our PA-BoVW model and the typical method for learning the visual words (denoted as 'Cluster' ), which trains a classification network on base classes and selects the clustering centroids of the feature vectors as visual words. Then we evaluate the performance our PA-BoVW optimized using only the image classification as the pretext task (denoted as 'Cls' ). Table 7 shows that the performance gains from 'Cluster' is smaller than that of our PA-BoVW using only the pretext task of image classification. Using the pretext task of PPC further boosts the performance substantially, which implies the importance of learning the embedding space by PPC in a self-supervised way. Distillation on deep features vs on BoVW representations. A classical way to perform knowledge distillation is to learn an independent feature space and perform distillation between two feature spaces. We conduct such experiment to compare between distillation on deep features and on BoVW representations. Specifically, we directly distill the pooled deep features in our trained PA-BoVW model to the feature space of the detector. The results of such method shown in Table 8 are much worse than our method. This is reasonable since distillation between feature space relies heavily on 1) the quality of the referenced features from which the distillation is performed and 2) well modeling of mapping between two feature space. By contrast, our method distills knowledge based on the similarity distribution over learned visual words, which benefits from similar merits of feature representations as Bag-of-Word representations used in NLP.  Qualitative Evaluation. Figure 3 (a) shows the t-SNE [29] visualization of proposal embeddings from randomly selected 30 instance bounding boxes per category. The baseline (TFA++ [38]) tends to misclassify some samples of novel classes as similar base classes. For instance, the samples from novel classes 'bird' and 'cow' cannot be clearly separated from other base classes like 'dog' and 'horse'. In contrast, applying our approach to the baseline model leads to more accurate boundaries. Figure 3 (b) shows the detection results of the baseline and our approach. We can observe that our method can successfully detect the novel objects while the baseline tends to misclassify these objects as base classes.

Conclusion
To solve the potential overfitting in few-shot object detection, we propose a knowledge distillation framework. We first learn a PA-BoVW model using two pretext tasks, namely Pixel-to-Propagation Consistency and image classification. Based on the PA-BoVW model, then we perform distillation to guide the learning of detector. As an orthogonal component, our approach can be easily combined with other methods and significantly improve the performance.

Supplementary Material for Few-Shot Object Detection by Knowledge Distillation Using Bag-of-Visual-Words Representations
A Results on Base Classes Table 1 shows the performance for base and novel classes on Novel Split 1 of PASCAL VOC dataset. Although AP for base classes (bAP50) is not our primary concern, our method makes competitive results. It can be observed that our method improves not only the performance of novel classes, but also the performance of base classes. These results demonstrate that our method can maintain the performance on previous knowledge without forgetting.

B Results over Multiple Runs
To eliminate the effect of sample variance introduced by the random selection of few-shot training samples, we fine-tune our model over 10 random selections of few-shot training samples independently for each experimental settings (including different novel splits and shot numbers), and obtain the average results on PASCAL VOC dataset. As shown in Table 2, our method improves the performance of TFA++ [38] under all settings.

C Comparison with More Baseline Methods
In Table 3 we integrate our method into two more baselines: TFA w/ fc [44] and Retentive R-CNN [9]. It can be observed that our method consistently boosts the performance, which shows the effectiveness of our method.

D More Ablation Studies
Effect of the number of visual words. Table 4 shows the effect of the number of visual words. It can be observed that the performance first improves rapidly with the increase of visual words and then starts to degrade after 256. This is mainly resulted from the limited size of data corpus for learning the visual words. Knowledge distillation vs initialization of the object detector vs multitask learning. We further explore other methods to learn a generalizable detector. As shown in Row 2 of Table 5, using the backbone pre-trained by PPC [51] to initialize the detector yields little improvement over the baseline. Row 3 shows that the performance degrades when using PPC for multi-task learning, presumably because PPC aims to distinguish between pixels, which is not entirely consistent with the objective of object detection.

E More Qualitative Detection Results
We provide more qualitative detection results under 10-shot setting of PASCAL VOC Novel Split1. As shown in Figure 1, our method reduces the appearance of each type of errors such as missing detections and misclassifying novel objects.