{'fraction_non_alphanumeric': 0.04491248753060669, 'fraction_numerical': 0.015144644962365103, 'mean_word_length': 4.657901487942534, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Weakly-supervised Any-shot Object Detection

Methods for object detection and segmentation rely on large scale instance-level annotations for training, which are difficult and time-consuming to collect. Efforts to alleviate this look at varying degrees and quality of supervision. Weakly-supervised approaches draw on image-level labels to build detectors/segmentors, while zero/few-shot methods assume abundant instance-level data for a set of base classes, and none to a few examples for novel classes. This taxonomy has largely siloed algorithmic designs. In this work, we aim to bridge this divide by proposing an intuitive weakly-supervised model that is applicable to a range of supervision: from zero to a few instance-level samples per novel class. For base classes, our model learns a mapping from weakly-supervised to fully-supervised detectors/segmentors. By learning and leveraging visual and lingual similarities between the novel and base classes, we transfer those mappings to obtain detectors/segmentors for novel classes; refining them with a few novel class instance-level annotated samples, if available. The overall model is end-to-end trainable and highly flexible. Through extensive experiments on MS-COCO and Pascal VOC benchmark datasets we show improved performance in a variety of settings.


Introduction
Over the past decade CNNs have emerged as the dominant building blocks for various computer vision understanding tasks, including object classification [3][4][5], detection [6][7][8], and segmentation [9,10]. Architectures based on Faster R-CNN [6], Mask R-CNN [9] and YOLO [8] have achieved impressive performance on a variety of core vision tasks. However, traditional CNN-based approaches rely on lots of supervised data for which the annotation efforts can be time-consuming and expensive [11,12]. While image-level class labels are easy to obtain, more structured labels such as bounding boxes or segmentations are difficult and expensive 1 . Further, in certain domains (e.g., medical imaging) more detailed labels may require subject expertise. This growing need for more efficient learning have motivated development of a variety of approaches and research sub-communities.
On one end of the spectrum, zero-shot learning methods require no visual data and use auxiliary information, such as attributes or class names, to form detectors for unseen classes from related seen category detectors [14][15][16][17]. Weakly-supervised learning methods [12,[18][19][20][21] aim to utilize readily available coarse image-level labels for more granular downstream tasks, such as object detection [16,17] and segmentation [12,22]. Most recently, few-shot learning [23][24][25][26] has emerged as a learning-to-learn paradigm which either learns from few labels directly or by simulation of few-shot learning paradigm through meta-learning [27][28][29]. However, all of the aforementioned sub-settings have been targeted, to a large extent, by different sub-community of researchers and algorithms. To date, no framework has been developed that can effectively scale to any amount of training data (from zero-shot to fully supervised), especially for granular instance-level visual understanding tasks.  Figure 1: Weakly-supervised Any-shot Detection. The data used in our setting is categorized in two ways: (1) image-level classification data for all the object classes, and (2) abundant detection data for a set of base object classes and limited (possibly zero) detection data for a set of novel object classes, with the aim to obtain a model that learns to detect both base and novel objects at test time.
We make two fundamental observations that motivate our work. First, image-level supervision is abundant, while instance-level structured labels, such as bounding boxes and segmentation masks, are expensive and scarce. This is reflected in the scales of widely used datasets where classification tasks have > 5K classes [5,30] while the popular object detection/segmentation datasets, like MSCOCO, have annotations for only 80 classes. A similar observation was made by Hoffman et al. [11] that introduced the LSDA model -a weakly-supervised object detection model that transformed imagelevel classifiers into object detectors. Second, the assumption that no instance-level supervision is available for target classes (as is the case for LSDA [11] and zero-shot methods [14][15][16][17]) is artificial.
In practice, it is often easy to collect few instance-level annotations and, in general, a good object detection/segmentation model should be robust and work with any amount of available instance-level supervision. Our motivation is to bridge weakly-supervised, zero-and few-shot learning paradigms to build an expressive and interpretable model that can operate and generalize with a type (weak/strong) and variety of instance-level supervision data (from 0 to 30+ instance-level samples per class).
In this work, we develop a weakly-supervised framework for object detection that scales with different levels of instance-level supervision ranging from no-data, to a few, to fully supervised (see Figure 1). The data used in our problem is categorized in two ways, (1) image-level classification data for all the object classes, and (2) abundant detection data for a set of base object classes and limited (possibly zero) detection data for a set of novel object classes, with the aim to obtain a model that learns to detect both base and novel objects at test time. We note that this setting shares some similarity with few-shot object detection [31][32][33], but assumes existence of image-level classification data for all classes; while also allowing us to operate effectively in the "zero-shot" setting.
Our algorithm, illustrated in Figure 2, jointly learns weak-detectors for all the object classes, from image-level classification data, and supervised regressors/segmentors on top of those for base classes (based on instance-level annotations in a supervised manner). The classifiers, regressors and segmentors of the novel classes are expressed as a weighted linear combination of its base class counterparts. The weights of the combination are determined by a multi-modal similarity measure: lingual and visual. The lingual similarity uses GloVe embeddings [34] of class labels, while for visual similarity we leverage meta-learning. The key insight of our approach is to utilize the multi-modal similarity measure between the novel and base classes to enable effective knowledge transfer and adaptation. The adopted novel classifier/regressors/segmentors can further be refined based on instance-level supervision, if any available. We experiment with the widely-used detection/segmentation datasets -Pascal VOC [35] and MSCOCO [1], and compare our method with state-of-the-art few-shot object detection and weakly-supervised object detection methods.
Contributions: Our contributions can be summarized as follows: (1) We study the problem of weaklysupervised object detection (image-level annotation) in light of limited detection/segmentation data ranging from no data (zero-shot) to a few (few-shot) supervised data regimes; (2) We propose a general, semantic and flexible end-to-end framework that can adopt classifiers/detectors/segmentors for novel classes by expressing them as linear combinations of their base class counterparts. In doing so, we leverage a learned multi-modal (lingual + visual) similarity metric. (3) We illustrate flexibility and effectiveness of our model by applying it to a variety of tasks (object detection and segmentation) and datasets (Pascal VOC [35], MSCOCO [1]); showing state-of-the-art performance. On MS-COCO, we get as much as +7.5/+10.4 mAP on detection/segmentation over the closest baseline [33].

Related Work
Few-shot object detection. Object detection with limited data was initially explored in a transfer learning setting by Chen et al. [36]. Lately, meta-learning [23][24][25][26]29] has emerged as a paradigm which attempts to resolve overfitting by simulating a learning-to-learn scheme with episodic tasks. In the context of object detection, Kang et al. [31] put forward a few-shot model where the learning procedure is divided into two phases: first the model is trained on a set of base classes with abundant data using episodic tasks, then in the second phase, a few examples of novel classes and base classes are used for fine tuning the model. Following this formulation, [32,33] employed better performing architecture -Faster R-CNN [6], instead of a one-stage detector YOLOv2 [8]. Yan et al. [33] also extended the problem formulation to account for segmentation masks in addition to detection. Conceptually closest to us is [32], where an approach for weight prediction mapping was learned. Our work adopts the two-phase learning procedure used in few-shot object detection [31][32][33]. However, we fundamentally differ in assuming that extra supervision in the form of image-level data over all the classes is available. Unlike [32], we learn a semantic mapping between weakly-supervised detectors and detectors obtained using a large number of examples.
Weakly-supervised object detection. Weak supervision in object detection takes the form of imagelevel labels, usually coupled with bounding box proposals [37,38], thereby representing each image as a bag of instances [18][19][20][21]39]. Bilen and Vedaldi [18] proposed an end-to-end architecture which softly labels object proposals and uses a detection stream in addition to classification stream to classify them. Further extensions followed, notably, Diba et al. [19] incorporated better proposals into a cascaded deep network, and Tang et al. [39] proposed an Online Instance Classifier Refinement (OICR) algorithm which refines predictions iteratively. More recently, further improvements were made by combining weakly-supervised learning with strongly-supervised detectors, by treating predicted locations from the weakly-supervised detector as pseudo-labels for a strongly-supervised variant [20,21]. In this work, we choose to adopt and build on top of single-stage OICR [39], hence enabling end-to-end training. However, our approach is not limited to the choice of weakly-supervised architecture. Notably, different from the weakly-supervised setup, our approach assumes abundant detection data for base classes and we compare and report performance on novel classes.
Our approach is inspired by LSDA [11]. LSDA assumes image-level supervision for all the classes and detection supervision for a subset of those classes (base). Their approach adapts image-level classifiers to object detectors by learning a transformation on base classes, with the aim to transfer this knowledge to novel classes with no detection data. In contrast, our model starts with weaklysupervised detectors, instead of classifiers, and considers a more generalized problem setting where we have varying degrees of detection supervision for novel classes ranging from zero to a few k-samples per class. As such, the setting in LSDA is simplified variant of our model with k = 0.
Zero-shot object detection. Zero-shot approaches rely on auxiliary semantic information to connect base and novel classes; e.g., text description of object labels or their attributes [14][15][16][17]. A common strategy is to represent all classes as prototypes in the semantic embedding space and to learn a mapping from visual features to this embedding space using base class data; classification is then obtained using nearest distance to novel prototypes. This approach was expended to detection in [40][41][42][43][44][45]. Bansal et al. [16], similarly, proposed method to deal with situations where objects from novel/unseen classes are present in the background regions. We too explore the setting where we are not provided with any instance data for novel classes, but in addition assume weak-supervision for novel object classes in the form of image-level annotations; such annotations readily available [30].

Problem Formulation
Here we formally introduce the weakly-supervised any-shot object detection / segmentation setup. We start by assuming image-level supervision for all the classes denoted by with |C| being number of object classes. We further extend the above image-level data with object-instance annotations by following the few-shot object detection formulation [31][32][33]. We split the classes into two disjoint sets: base classes C base and novel classes C novel ; C base ∩ C novel = ∅. For base classes, we have abundant instance data  Figure 2: Overall Architecture. We form detectors/segmentors of base classes as a refinement on top of the weak detectors. The detectors/segmentors of novel classes utilize a similarity weighed transfer from the base class refinements. In k-shot setting, (few) novel class instance annotations are incorporated through direct adaptation of the resulting novel detectors/segmenters through fine-tuning. The similarity is a combination of lingual and learned visual similarity (pink boxes). All detectors are built on top of Faster/Mask RCNN architecture which comprises of classification and regression heads with shared backbone (in cyan) and simultaneously trained region proposal network (RPN).
novel classes, we have limited instance data D novel = {(x i , c i , y i )}, where a k-shot detection / segmentation task only has k bounding boxes / masks for each novel class in C novel . Note, for weakly-supervised zero-shot k = 0 and D novel = ∅.

Approach
We propose a single unified framework that leverages the weak image-level supervision for object detection / segmentation in any-shot setting. That is, our proposed approach can seamlessly incorporate arbitrary levels of instance-level supervision without the need to significantly alter the architecture.
Our proposed meta-learning framework builds upon the Faster R-CNN [6] / Mask R-CNN [9] architecture. Faster R-CNN [6] utilizes a two-stage pipeline in order to perform object detection. The first stage uses a region proposal network (RPN) to generate class-agnostic object region proposals {rbox i,j } for image i. The second stage is a detection network (Fast R-CNN [46]) that performs RoI pooling, forming feature vector z i,j = RoIAlign(x i , rbox i,j ) for proposal j, and learns to classify this RoI feature vector z (we drop proposal and image indexing for brevity for remainder of the section) into one of the object classes and refine the bounding box proposals using a class-aware regressors. Conceptually, an R-CNN object detector can be thought of as a combination of a classifier and regressor (see Figure 2). Mask R-CNN [9] is a simple extension to the Faster RCNN framework, wherein an additional head is utilized in the second stage to predict the segmentation masks. Figure 2 details the proposed architecture. The model consists of two branches: i) the weaklysupervised branch that trains detectorsĉ = softmax(f W weak (z)) using image-level supervision D class , and ii) a supervised branch that uses detection data D base /D novel to learn a refinement mapping from the weak detector to category-aware classifiers, regressors, and segmentors f W * (z); * ∈ {cls, reg, seg} used in the second stage of Faster / Mask R-CNN. Note that weak detectors simply output proposal box of the pooled feature vector as locationŷ = pbox; while refined detectors are able to regress a better box. Here f W (·) is a learned neural network function parametrized by W. We jointly train both branches and RPN, and since our approach follows the meta-learning paradigm, learning is divided into two stages: base training and fine-tuning 2 .
Base training: During base training, instances from D base are used to obtain a detector / segmentation network for the base classes C base . Specifically, for each b ∈ C base , we form the category-aware classifier and regressor for the base classes as additive refinements to the corresponding weak counterparts. For region classifiers this takes the form of:ĉ = arg max where f ∆W cls base (z) is a zero-initialized residual to the logits of the weakly supervised detector. The object location is similarly refined:ŷ = pbox + f W reg base (z).
(2) The segmentor can be define analogously. Please see Appendix A for details.
Novel fine-tuning (k > 0): In the fine-tuning phase, the detectors / segmentors of the base classes are used to transfer information to the classes in C novel . The network is also fine-tuned on D novel , which, for a value of k, contains k bounding boxes / masks for novel and base classes. Here we consider the case of k > 0; we later address k = 0 case, which does not require fine-tuning. The key insight of our approach is to use additional visual and lingual similarities between the novel and base classes to enable effective transfer of the network onto the novel classes under varying degrees of supervision. For a specific proposal pbox with features z, let S(z) ∈ R |C novel |×|C base | denote similarity between base classes and novel classes. The dependence on z stems from visual component of the similarity and is discussed in Section 4.2. Given this, for each proposal z, the category-aware classifier for the novel classes is obtained as follows:ĉ = arg max instance-level transfer from base classes where S(z) = softmax(S lin S vis (z)), and denotes broadcast of vector similarity S vis (z) ∈ R |C base | followed by element-wise product with lingual similarity S lin ∈ R |C novel |×|C base | . The interpretation of Eq. (3) is actually rather simple -we first refine the weak detectors for novel classes by similarity weighted additive refinements from base classes (e.g., novel class motorbike may relay on base class bicycle for refinement; illustrations in Appendix F.), denoted by "instance-level transfer from base classes" 3 ; we then further directly adapt the resulting detector (last term) with few instances of the novel class. Similarly, for each z, the novel class object regressor can be obtained as, instance-level transfer from base classes Again, for the segmentation head f W seg novel (z) the formulation is identical to Eq.(4). In the following sections we describe the individual elements in more detail.
Weakly-supervised zero-shot (k = 0): As we mentioned previously, our model is also readily applicable when C novel = ∅. This is a special case of the formulation above, where fine-tuning is not necessary or possible, and we only rely on base training and apply novel class evaluation procedure. The predictions for novel classes can be done as in Eq.(3) and Eq.(4) but omitting the "instance-level direct adaptation" term in both cases.

Weakly-Supervised Detector
As mentioned earlier, our approach leverages detectors trained on image level annotations to learn a mapping to supervised detectors. We highlight that our approach is agnostic to the method used to train the weakly-supervised detector, and most of the existing approaches [18,20,39] can be integrated into our framework. We, however, use the Online Instance Classifer Refinemnet (OICR) architecture proposed by Tang et al. [39] due to its simple architecture. OICR has R "refinement" modules f W weak r (z) that progressively improve the detection quality. These individual "refinement" modules are combined to obtain the final prediction as follows, We use the same loss formulation L weak (a,â) described in [39], which compares predicted (â) and ground truth (a) class labels, to train the OICR module (see Sect. 4.3). Additional details are in [39].

Similarity Matrices
As described in Eq. (3) and (4), the key contribution of our approach is the ability to semantically decompose the classifiers, detectors and segmentors of novel classes into their base classes' counterparts. To this end, we define a proposal-aware similarity S(z) ∈ R |C novel |×|C base | , where each element captures the semantic similarity of novel class n to base class b. We assume the similarity matrix S(z) can be decomposed into two components: lingual S lin and visual S vis (z) similarity.
Lingual Similarity. This term captures linguistic similarity between novel and base class labels. The intuition lies in the observation that semantically similar classes often have correlated occurrences in textual data. Therefore, for a novel class n and a base class b, S lin n,b = g n g b , where g n and g b are 300-dimensional GloVe [34] vector embeddings for n and b respectively 4 .
Visual Similarity. We propose the Meta-Visual Projection Network (VPN) that aims to learn interclass similarity by using image representations. We form k-shot base meta-learning tasks from D base ; this ensures that the visual similarity can be learned from only base classes and is applicable in the case where k = 0 for novel classes in Eq. (3) and (4). A k-shot task consists of a support set The support set M τ contains k images for each class, where each image is annotated with an object bounding box / mask belonging to the corresponding class. The query set Q τ contains images with annotations for evaluation. Based on the support set, we define v b ∈ R 2048 as the mean class-attentive vector [33] for base class b, where, where RoIAlign(.) computes the RoI features corresponding to the ground-truth bounding box region for an input image. This is equivalent to passing the images through the backbone and setting the ground-truth bounding box as the proposal. A is a channel-wise soft-attention layer. The resulting v b is effectively a learned embedding for the base class b. For every proposal z i,j in the query set one can then measure its similarity, to each base class-attentive vector embedding, with implicit goal that if a proposal contains an object of base class b, it should be closer to v b than to any other class embedding v ¬b . P and A are learned during the base-training phase along with other parameters of the full model. For each proposal z i,j , we set c i,j to the class of the ground-truth bounding box with which z i,j has the highest overlap (in terms of IoU). In case z i,j doesn't overlap with any bounding box among the base classes, c i,j is set to the background class. P and A are then trained by minimizing the regularized max-margin loss: where ¬c i,j is the set c i \ c i,j , and c i = {c i,j }, ∀j. See Sec. 4 for how we combine S vis (·) with S lin .

Training
We now describe the optimization objective used to train our proposed approach in an end-to-end fashion. As mentioned earlier, due to the meta-learning nature of the task, we use separate objectives during base training and fine-tuning. During base training, the objective can be written as, where L rcnn is the Faster/Mask R-CNN [6,9] and L weak is the OICR [39] objective; α = 1, β = 1 are weighting hyperparameters. In fine-tuning, we refine the model only using L rcnn . Fine-tuning only effects last term of Eq. (3) and (4), while everything else is optimized using base training objective. Further implementation details are in Appendix B; code will be released shortly.

Comparison to Few-shot Object Detection
Datasets. We evaluate our models on VOC 2007 [2], VOC 2012 [35], and MSCOCO [1] as used in the previous few-shot object detection works [31][32][33]. We consistently follow the data splits  introduced and used in [31,33]. In case of VOC, we use VOC 07 test set (5k images) for evaluation and VOC 07+12 trainval sets (16.5k images) for training. The 20 object classes are divided into 15 base and 5 novel classes with 3 different sets of class splits. For novel classes, we use images made available by Kang et al. [31] for k-shot fine-tuning where k ∈ {0, 1, 2, 3, 5, 10}. We report mean Average Precision (mAP) on novel classes and use a standard IoU threshold of 0.5 [2].
Similarly, for the MSCOCO [1] dataset, consistent with [31] we use 5k images from the validation set for evaluation and the remaining 115k trainval images for training. We assign 20 object classes from VOC as the novel classes and remaining 60 as the base classes. The k-shots tasks are sampled as before with k ∈ {0, 10, 30}, and we follow the standard evaluation metric on COCO [6].
Baselines. Our setting is new. We compare our approach to the closest state-of-the-art few-shot object detection methods [31][32][33]. Notably, [33] is most similar with respect to the architecture and class-attentive vector formulation. We also adapt the baselines that they train on the task, namely Joint and Transfer, where Joint refers to learning detectors together for all classes on abundant base data and few-shot novel data, and Transfer refers to learning detectors for base classes from the abundant data, and then fine-tuning it on few-shot novel data to obtain 5 novel class detectors.
PASCAL VOC. Table 1 summarizes the results on VOC for three different novel class splits with different k-shot settings. We use Faster R-CNN [6] with ResNet-101 [4] as the backbone which is pre-trained on ImageNet-1k [3]. Our approach outperforms the related state-of-the-art methods on all values of k including no instance-level supervision for novel classes (k = 0) showing the effectiveness of transfer from base to novel classes, while additionally using only weak image-level supervision for all classes which is readily available and much cheaper to obtain [13]. Our improvements come mainly from structured transfer from base classes, applied on top of the weak detectors. Table 2 describes the results on COCO dataset. Here we use Faster R-CNN [6] with ResNet-50 [4] as the backbone. We observe similar trends as above. In addition, we note that our performance consistently increases with the value of k showing that our approach is effective and flexible in scaling with the degree of instance-level supervision ranging from zero to a few. Figure 3 shows some qualitative results, indicating our method is able to correctly detect novel classes.

MS-COCO.
Ablation. A complete ablation study is provided in Appendix E. We report performance for all novel splits starting with only weak detectors and progressively adding the terms in Eq.(1), (3) and (4). Top-K transfer from base classes (instance-level transfer) is used as a baseline (akin to LSDA [11]). Weighting with lingual similarity results in 5.3. mAP improvement; combination with visual is an additional 0.9 mAP. Finally, transfer from base classes for regressors leads to added gain of 12.9.   [18,20,21,39].
In addition, we assume instance-level supervision for base classes in the dataset, and we report performance on the novel classes; no instance-level supervision is given for novel classes.
Baselines. We report results of related baselines [18,39]. Comparison to OICR [39] is most meaningful as this is the weakly-supervised branch in our model. We report figures from both the published [39] and our re-implemented version of OICR, and we also include latest state-of-the-art method [20] in weakly-supervised object detection. For an extensive list, please see Table 1 in [20]. Results. Table 3 provides a summary of the results.
Here we use pre-trained proposal network (EdgeBox [38]) instead of jointly trained RPN for fair comparison. We outperform the related OICR [39] method, on which we build, on each novel class despite the fact that our re-implementation didn't reach the published performance 5 . We highlight that the relative improvement with respect to our reimplementation of OICR is a significant 59%. Our results beat latest state-of-the-art method of [20] on 3 out of 5 novel categories. Our marginally inferior average performance with respect to [20] is dominated by sofa which performs worse in our reimplemented version of OICR. We note that our approach is agnostic to the model architecture used for weak-supervision and can be further improved (e.g., if built on top of [20]). Further, our model benefits significantly when trained with more data and not restricted to pre-trained proposals; our result on combined VOC 2007 + 2012 with jointly trained RPN can achieve 68.9 mAP (see Table 1, Novel Set 1). Table 4 summarizes the results. We choose ResNet-50 [4] as the backbone and extended our model with an additional head to enable segmentation. Our approach consistently improves over [33], demonstrating that our approach is not limited to bounding boxes and is able to generalize over the type of downstream structured label, including segmentation masks. Find full table in Appendix D.

Discussion and Conclusion
We propose an intuitive weakly-supervised model that is applicable to a range of supervision: from zero to a few instance-level samples per novel class. For base classes, our model learns a mapping from weakly-supervised to fully-supervised detectors/segmentors. By leveraging similarities between the novel and base classes, we transfer those mappings to obtain detectors/segmentors for novel classes; refining them with a few novel class instance-level annotated samples, if available. This versatile paradigm works significantly better than traditional few-shot or weakly-supervised detection. We omitted details of the segmentation from the main paper due to lack of space; we describe them here. Our implementation of segmentation module can be seen an extension to the Fast R-CNN [46] pipeline described in Section 4 of the main paper. In particular, the segmentation module consists of a transposed-convolution layer (nn.ConvTranspose2D), followed by ReLU, and a 1 × 1 convolution layer (nn.Conv2D). The feature vector z i,j for a proposal j in image i is of dimension

Acknowledgments and Disclosure of Funding
where 2048 is the number of channels and 7 is the spatial resolution of the proposal's feature map. The segmentation module upsamples z (as in the main paper we drop i, j indexing) using the transposed convolution layer with a kernel size of 2, and then produces a class-specific mask using a 1 × 1 convolution layer. The resulting mask output is of size (|C| × 14 × 14), where C is the total number of object classes.
As in the main paper, C is a union of base, C base , and novel, C novel , classes. For the base classes the refinement process is analogous to the regression described in Eq.(2) of the main paper. Notably, The formulation for the novel classes, which includes the transfer from the base classes and direct adaptation (when k > 0), can similarly be formulated analogus to Eq.(4) in the main paper: instance-level transfer from base classes Here f W seg * (·) is the class-specific output of the segmentation module obtained after the 1 × 1 convolution. During training, we use the same loss formulation for L mask as described in [9], where a per-pixel binary cross-entropy loss is used. During inference, the mask is interpolated to fit the regressed proposal (as obtained by Eq.(2) and Eq.(4) in the main paper) to produce the final output. For the weakly-supervised zero-shot (k = 0) scenario, the predictions for novel classes can be done as in Eq. (11) but omitting the "instance-level direct adaptation" term.

B Implementation Details
For base-training, we train our model jointly with weak-supervision and base-detection/-segmentation losses with equal weighting (see Section 4.3). In particular, we use image-level data for all the classes to train the weakly-supervised OICR [39] branch, and use detection/segmentation data of base classes for training base detectors/segmentors. The proposals used for training weakly-supervised branch come from the RPN trained using the base-detection branch. We employ task-based training procedure where each task consist of 5 support images per class and 8 query images in a batch for base-training. We use 4 Nvidia Tesla T4 GPUs to train models. We build on top of Detectron2 [47] library written in PyTorch [48] framework, and unless mentioned, we keep their default configuration: SGD as the optimizer, RoI Align [9] as the pooling method, ResNet layer sizes/parameters. We use the standard loss for Faster R-CNN, i.e., cross-entropy loss for classifier and smooth-L1 loss for regressor as described in [6].
Few-shot Object Detection: VOC. We train on VOC 07 + 12 dataset. We use a learning rate of 0.02 over 30K iterations. We decrease the learning rate by a factor of 10 at 12K and 24K iteration.
For fine-tuning, we are given k-shot data for novel classes where k ∈ {1, 2, 3, 5, 10}. We linearly scale the number of SGD steps for optimizing over the k-shot data. In particular, we choose 50 iterations for k = 1, 100 iterations for k = 2, and similarly linearly scale to 500 iterations for k = 10.
Few-shot Object Detection: COCO. In the case of COCO dataset, we instead use 270K iterations (default in Detectron2 [47]) to account for more data as compared to VOC. For fine-tuning, we use 500 iterations for 10-shot and 1500 iterations for 30-shot experiment.
Weakly-supervised Object Detection Here we use a pre-trained VGG-16 [49] as the backbone to be consistent with the prior state-of-the-art works [18,20,39]. We use a learning rate of 0.005 over 40K iterations for optimization.

C Comparison to Few-shot Object Detection with Annotation Budget
Our problem setting is similar to few-shot object detection, as formulated in [31][32][33]. However, we assume additional availability of weak image-level supervision for novel classes. We argue this is a reasonable assumption considering that such annotations are readily available in abundance for thousands of object classes (∼ 22K in ImageNet [50] and ∼ 20K in Open Image v4 dataset [30]). However, this raises an interesting question as to what form of supervision maybe more valuable, if one is to collect it. To experiment with this, we conceptually fix the annotation budget and compare performance of state-of-the-art published few-shot detection methods [31][32][33] in a 10-shots setting with our weakly-supervised zero-shot (k = 0) detection setting with 120 weak image-level annotations for each of the novel classes 6 . The 12× conversion factor between object instance labels and weakly-supervised image-level labels is motivated by timing reported in [13]. While this is not a rigorous experiment, it does allow us to compare performance of our and few-shot object detection state-of-the-art methods under a presumed fixed annotation budget. Performance is reported in Table 5 below. Observations in Table 5 indicate that our approach, that uses only weak-supervision for novel classes, performs superior on average, by a margin of 4.1 mAP, to the best few-shot detection variant. It perfroms better on 3 out of 5 object classes, with [31] doing best on the remaining ones. These results suggests that weak-labels, in fact, may contain more information/be more valuable than instance-labels obtained using the same budget of annotation time. Notably, the former are also easier to collect and tend to be less ambiguous. In practice, we posit that both forms of annotation are useful. The optimal ratio of the two would likely be dictated by the specific problem setting. Our model, while does not require instance-level supervising for novel classes, is specifically designed to operate in the heterogeneous data annotated regime; unlike [31][32][33]. In the future, it would be interesting to study behaviour of object detection / segmentation as a function of different ratios of weak image-level and strong instance-level object annotations.

D Comparison to Few-shot Instance Segmentation
As described in Section 5.3 of the main paper, we analyse the performance of our proposed approach on the task of Few-shot Instance Segmentation. To ensure a fair comparison to the baselines, we choose ResNet-50 [4] as the backbone and use an additional segmentation head (as described in Section A of the supplementary). The k-shot tasks are sampled following [33] with k ∈ {0, 5, 10, 20}, and we follow the standard evaluation metrics on COCO [9]. The complete results are shown in Table  6. Our approach consistently improves over [33], demonstrating that our approach is not limited to bounding boxes and is able to generalize over the type of downstream structured label, including segmentation mask.
As can be seen from Table 6 we get a significant boost in performance with k = 5. However, the improvement obtained by going to k = 20 is comparatively less significant. This is consistent with observations made in [33].

E Ablation study
Please refer to Section 5.1 of the main paper for a detailed explanation of task setup. We perform ablation over the terms used in Equations (3) and (4) of the main paper on all three novel class splits for VOC 07 + 12 dataset. The results are summarized in Table 7. In particular, we start by forming detectors using only the weakly-supervised branch f W weak (denoted as "weak" in Table  7), and progressively add refinement terms to observe their impact on detection performance. Note that we do not use any instance-level annotations for the novel classes, i.e., we report results for weakly-supervised zero-shot setting (k = 0). We then incorporate the transfer from the base classes f ∆W cls base into the weak detector (see Equation  3 in the main paper). For each novel class, we first compare to the two baseline approaches: averaging over all the base classes (denoted by weak + avg(∆)), and averaging over top-5 most similar classifiers (denoted by weak + top-5(∆)). For each novel class, similar to LSDA [11], the top-5 most similar classifiers are computed using the inner-product between the weights of f ∆W cls novel and f ∆W cls base . We note that top-5 (row 3) performs better than naïve averaging (row 2), which shows that an informed similarity measure between base and novel classes leads to better performance.
We then explore the role of proposed similarity matrices, detailed in Section 4.2 of the main paper. The similarity matrix between base and novel classes can be decomposed into two components: lingual similarity S lin and visual similarity S vis (z). We analyse the impact of using the aforementioned similarities in obtaining category-aware classifiers and regressors. Following the terms in Eq. (3) and (4) of the main paper, we define ablated variants of our final model. "weak + S lin cls " is the model where-in the category-aware classifier for the novel classes is obtained by using only the lingual similarity, and the category-aware regressor is fixed to predict zeros (i.e. the model uses the output of the category-agnostic Fast-RCNN regressor pbox). Similarly, "weak + S lin cls + S vis cls " is defined as the model where-in the category-aware classifier for the novel classes is obtained by using both lingual and visual similarities, and the category-aware regressor is fixed to predict zeros. Finally, in order to understand the impact of lingual similarity on both the category-aware classifiers and regressors, we define "weak + S lin " as the model that uses only lingual similarity in Eq.(3) and (4) of the main paper. The "Final Model" in Table 7 uses both similarities to obtain category-aware regressors and classifiers for the novel classes. The ablation clearly highlights importance of all terms in our model. Figure 6 provides qualitative examples to further highlight the impact of using our proposed transfer from base to novel classes. Column (a) in Figure 6 refers to "weak", column (b) refers to "weak + S lin cls + S vis cls ", column (c) refers to the "Final Model" with k = 0, and column (d) refers to the "Final Model" after being trained with k = 10 shots. It can be seen that the "weak" model either fails to identify all objects or doesn't generate high-probability proposals for the desired objects (column (a)). "weak + S lin cls + S vis cls " improves performance by generating a bunch of reasonable proposals (column (b)). The "Final Model" further refines the proposals to obtain accurate bounding boxes for F Analysis of Similarity Matrices Figure 4: Normalized lingual similarity matrix for the second novel split in PASCAL VOC. Note that S lin is proposal-agnostic. Most of the similarities are intuitive and semanticsofa is most similar to a chair; horse to a dog and a sheep; cow is similar to a sheep; aeroplane is related to other transportation vehicles like car and boat. A notable departure is a bottle which has no closely related categories among base classes, resulting in less interpretable similarity and transfer.
As described in Section 4 of the main paper, the key contribution of our approach is the ability to semantically decompose the classifiers, detectors and segmentors of novel classes into their base classes' counterparts. To this end, we define a proposal-aware similarity S(z) ∈ R |C novel |×|C base | , which is further decomposed into two components: lingual S lin and visual S vis (z) similarity. Please refer to Section 4.2 of the main paper on details pertaining to how these similarities are computed.
We qualitatively visualize these similarity matrices to highlight the intuitive semantics learned by our proposed model. Figure 4 shows the normalized lingual similarity matrix S lin ∈ R |C novel |×|C base | for the second novel split in PASCAL VOC. Figure 5 shows the normalized visual similarity S vis (z) ∈ R |C base | for each proposal z (highlighted in blue).

G Few-shot Performance on VOC's Base Classes
Due to lack of space, in the main paper, we focus on the detection/segmentation results obtained on the novel object classes; however, our model also learns to detect/segment base class objects as well. We now illustrate that our proposed method not only improves performance on novel classes, but also consistently outperforms baselines on the base classes. The experimental setup and baselines are identical to the one described in Section 5.1 of the main paper. Table 8 summarizes results on VOC [2] for the 1-st novel split with k-shots, k ∈ {3, 10}.
Our approach outperforms the related state-of-the-art methods on both novel and base classes. It is important to note that we are not using any additional annotations for the base classes (as compared to [31] and [33]). The significantly better performance on the base classes can be mainly attributed to the structured decomposition of our detectors: weak detector + learned refinement. We believe that such a decomposition results in an inductive model bias that leads to convergence to an ultimately better solution. In other words, these results suggest, that such decomposition may potentially be useful even in the traditional purely supervised setting.

H Additional Visualizations on MSCOCO Detection and Segmentation
We show additional visualizations highlighting the performance of our approach on the MSCOCO [1] dataset. The experimental setup is identical to the ones described in Sections 5.1 (for detection) and 5.3 (for segmentation) of the main paper. Figure 7 shows additional examples for the task of object detection, and Figure 8 shows additional examples for the task of instance segmentation. Note that these visualizations are generated on novel classes under the k = 0 setup.
(a) Complementary to S lin that assigns weights to classes boat and car, S vis (z) is additionally able to capture that an aeroplane flying in the sky shares some visual characteristics with the class bird.
(b) Complementary to S lin that gives a large weight to the class chair, S vis (z) is additionally able to capture that there is a high correlation between the class person and the class sofa. This follows the common observation that people are likely to be sitting on a sofa.
(c) Complementary to S lin that gives a large weight to the class sheep, S vis (z) is additionally able to capture that the class cow is visually similar to other animal classes bird, cat, and dog.   Figure 6: Qualitative Visualizations for the Ablation Study. (a) refers to the "weak" model, (b) refers to "weak + S lin cls + S vis cls ", (c) refers to the "Final Model" with k = 0, and (d) refers to the "Final Model" after being trained on k = 10 shots. Appendix E provides a detailed description of these models.