{'fraction_non_alphanumeric': 0.03131404851472305, 'fraction_numerical': 0.014684135426125308, 'mean_word_length': 4.669363141638477, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 4, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Better Generalized Few-Shot Learning Even Without Base Data

This paper introduces and studies zero-base generalized few-shot learning (zero-base GFSL), which is an extreme yet practical version of few-shot learning problem. Motivated by the cases where base data is not available due to privacy or ethical issues, the goal of zero-base GFSL is to newly incorporate the knowledge of few samples of novel classes into a pretrained model without any samples of base classes. According to our analysis, we discover the fact that both mean and variance of the weight distribution of novel classes are not properly established, compared to those of base classes. The existing GFSL methods attempt to make the weight norms balanced, which we find helps only the variance part, but discard the importance of mean of weights particularly for novel classes, leading to the limited performance in the GFSL problem even with base data. In this paper, we overcome this limitation by proposing a simple yet effective normalization method that can effectively control both mean and variance of the weight distribution of novel classes without using any base samples and thereby achieve a satisfactory performance on both novel and base classes. Our experimental results somewhat surprisingly show that the proposed zero-base GFSL method that does not utilize any base samples even outperforms the existing GFSL methods that make the best use of base data. Our implementation is available at: https://github.com/bigdata-inha/Zero-Base-GFSL.


Introduction
Few-shot learning (FSL) (Fei-Fei, Fergus, and Perona 2006;Lake et al. 2011;Lake, Salakhutdinov, and Tenenbaum 2015) is now a major problem motivated by the practical difficulty of data collection. Trying to mimic the human's ability to acquire general knowledge with a few observations, the goal of FSL is to learn new knowledge by training only a few samples on a model, which is often a pretrained model. Many existing works focus on a scenario where the resulting model only discriminates novel classes, referred to as standard fewshot learning (Finn, Abbeel, and Levine 2017;Koch et al. 2015;Snell, Swersky, and Zemel 2017;Vinyals et al. 2016). It is more challenging yet more practical when the model can infer both existing classes (a.k.a. base classes) already trained in the model and unseen classes (a.k.a. novel classes) newly learned with a few samples. Only some recent works (Gidaris  Shi et al. 2020) tackle this version of FSL, namely generalized few-shot learning (GFSL), which not only aims to learn about novel classes but also preserve the existing knowledge for base classes.
Due to its simplicity and performance, transfer learning becomes a dominant approach in GFSL, where we fine-tune a model already trained over base classes to additionally learn the knowledge of new classes with a few samples. Despite the small volume of new data, the fine-tuned model gets easily biased to novel classes to the point that the model becomes pretty useless for base classes. Hence, most existing works in GFSL focus on how to make the resulting model well balanced over base and novel classes by performing balanced fine-tuning (Kukleva, Kuehne, and Schiele 2021;Li et al. 2019;Qi, Brown, and Lowe 2018) or leveraging additional architectures (Gidaris and Komodakis 2018;Ren et al. 2019;Yoon et al. 2020) and supplement information (Li et al. 2020;Shi et al. 2020).
All the aforementioned approaches assume the presence of base data (i.e., samples of base classes) for preserving the knowledge of base classes as well as learning the relationship between novel and base classes. In reality, however, base data may not be always available due to some privacy or ethical issues. For instance, Google releases the highly generic model BiT (Kolesnikov et al. 2020) of 18,000 classes, but its training data JFT (Sun et al. 2017) consists of millions of private images that should not be exposed to public. Furthermore, retraining the base data has never been a perfect solution for GFSL either. In spite of retraining overhead, the overall performance can be highly dependent on which base samples are selected. To be shown by our experimental results, the state-of-the-art GFSL methods using base data turn out to be even less accurate than our GFSL method without employing any base samples.
Beyond the limitation of the existing GFSL approaches, this paper focuses on an even more challenging scenario of few-shot learning, called zero-base GFSL, where we are free to use a pretrained model somehow learned over base classes but cannot retrain any of base samples during GFSL. Obviously, fine-tuning a pretrained model with only novel samples leads to the model highly biased toward novel classes even if we freeze the feature extractor of the model. Thus, even in the frozen feature space built on base classes, novel samples are capable of forming undesirably large decision boundaries, which is hard to be addressed without jointly training all of the base and novel samples. Figure 1(b) shows such an example where some base samples, which used to be well discriminated in their pretrained model (Figure 1(a)), can be misclassified as a novel class with a large decision boundary. This leads to the model pretty inaccurate for base classes without normalization as shown in Figure 2.
Then, how could only a few novel samples make the decision boundaries highly biased toward novel classes in the feature space? To answer this question, we investigate the distribution of weights of classifiers that actually determine decision boundaries, and discover the following undesirable facts in terms of its mean and variance. First, there is an imbalance between the variances of weight distributions of base and novel classes, which we also find is related to the norm imbalance tackled by many existing GFSL methods (Fan et al. 2021;Gidaris and Komodakis 2018;Qi, Brown, and Lowe 2018;Wang et al. 2020). However, these methods do not consider how the means of weight distributions are different between base and novel classes. This paper newly observes that the average weight of the novel classifier is positively shifted from that of the base classifier, which we call mean shifting phenomenon. Although this phenomenon is indeed a more crucial reason behind the biased model, existing works rarely try to fix the shifted mean of novel classes.
To remedy these undesirable mean and variance of novel classes, this paper designs a new normalization method that can achieve a centered mean as well as a balanced variance without retraining any base samples. As experimentally shown in Figure 1(c), our normalization method enables the classifier to form well-balanced decision boundaries for both base and novel classes, and thereby it is observed in Figure  2 that the performance degradation of base classes is prevented while keeping the performance of the novel classes as high as possible. With only 5-shot of novel classes, we can keep the performance of both base and novel classes close to their upper bounds that are conditional accuracies given only base and novel classes, respectively. In our experiments, without using any base data, our method even beats the existing state-of-the-art GFSL methods with clear margins, like 4.59% and 3.53% 5-shot accuracy in miniand tiered-ImageNet, respectively.

Related Works
Few-shot learning. Few-shot learning (FSL) has been studied mostly for the standard scenario aiming to learn novel classes without having to preserve the existing knowledge. The recent methods of standard FSL can be divided into the following two categories: meta learning and transfer learning. Meta learning methods commonly train a metalearner in an episodic way, where the base data is divided into multiples tasks each with a few samples and thereby class-agnostic knowledge is acquired from previous experiences (Finn, Abbeel, and Levine 2017;Jamal and Qi 2019;Lee et al. 2019;Sun et al. 2019;Snell, Swersky, and Zemel 2017;Vinyals et al. 2016;Xu et al. 2021;Zhang et al. 2020). Unlike meta learning, which requires a complex episodic training, transfer learning (Chen et al. 2019;Dhillon et al. 2020;Tian et al. 2020;Wang et al. 2019) simply re-uses the feature extractor learned from all collected base samples when fine-tuning only the classifier of the model being trained for FSL. These transfer learning approaches often show a better performance than meta learning particularly in deeper neural networks (Dhillon et al. 2020), and therefore are getting more attention from the state-of-the-art methods. Although transfer learning seems to be quite effective in standard FSL, its performance is still unsatisfactory in generalized few-shot learning (GFSL) without leveraging additional techniques. However, this paper claims that our proposed normalization method can enable simple transfer learning to be highly effective in GFSL even without using any base samples. Generalized few-shot learning. GFSL aims to handle both base and novel classes in a joint space, and hence is regarded to be more difficult than standard FSL. In GFSL, we not only need to learn as much new knowledge as possible but also preserve the pretrained knowledge over base classes. The major approach to this end in GFSL is not to entirely change a pretrained model as we do in transfer learning, but to train some extra architectures, which inform the original output of the pretrained model to be properly modified for a balanced inference between base and novel classes (Gidaris and Komodakis 2018;Ren et al. 2019;Yoon et al. 2020) occasionally by leveraging supplement information (Li et al. 2020;Shi et al. 2020).
The other approach without introducing any extra architecture is to fine-tune a pretrained model with a balanced dataset of base and novel classes, namely balanced fine-tuning. In order to improve the performance, balanced fine-tuning is often performed together with additional techniques like weighting imprinting of classifiers (Qi, Brown, and Lowe 2018) and introducing a three-step framework (Kukleva, Kuehne, and Schiele 2021). For a better balanced dataset, hallucination approaches (Hariharan and Girshick 2017;Wang et al. 2018) synthesize novel instances based on the base dataset. Incremental few-shot learning. In continual learning, GFSL is being extended to incremental few-shot learning (IFSL) (Kukleva, Kuehne, and Schiele 2021;Mazumder, Singh, and Rai 2021;Tao et al. 2020;Zhang et al. 2021), in which the model has to go through a series of tasks of few-shot classes. Similar to the existing GFSL methods, most IFSL methods take the best use of the previously collected dataset to preserve the previous knowledge and learn the relationship among different classes.
All the existing works in GFSL and IFSL somehow need base data for either training extra architectures, balanced finetuning, or preserving the previous knowledge. To our best knowledge, this paper is the first study that resolves zero-base GFSL via weight normalization without using base data. Although the existing GFSL methods (Fan et al. 2021;Gidaris and Komodakis 2018;Qi, Brown, and Lowe 2018;Wang et al. 2020) also perform basic weight normalization using a cosine classifier, their normalization scheme alone fails to achieve a satisfactory performance without their additional training techniques relying on base data. In this paper, we investigate what occurs to weight distribution during GFSL, and thereby effectively overcome the limitation of the existing basic normalization method.

Preliminary Basic Framework of GFSL
In generalized few-shot classification, the training dataset is composed of a base dataset, denoted by , where (x, y) is a training instance and K base K novel is usually assumed. We denote C base and C novel be two disjoint sets of base classes and novel classes, respectively, where |C base | |C novel |. Then, the goal of GFSL is to train a model Φ = {φ, θ} with D base ∪ D novel such that Φ can discriminate any classes in C base ∪ C novel , which is different from standard FSL aiming to classify only C novel . φ and θ are parameters of the feature extractor and the linear classifier of the model, and we also denote f φ (x) be the feature vector returned from φ given x, where the dimensionality of each feature vector is denoted as d. We then make inference by taking the softmax of θ f φ (x) for a given input sample x.
The basic framework of GFSL consists of two stages, namely pre-training and fine-tuning. This two-stage transfer learning scheme, which fully utilizes the knowledge of base classes to learn few-shot novel classes, is regarded as one of the leading paradigms for GFSL as well as standard FSL due to its simplicity and effectiveness. Pre-training. In the pre-training stage, we train a full model , and R is the regularization term for the full model. This stage is equivalent to typical supervised learning as D base has a sufficient number of samples. Fine-tuning. Once a model is well-trained over base classes, we then fine-tune the model with respect to C base ∪ C novel . In order to incorporate novel classes, the model should be extended to Φ = {φ, θ} in this fine-tuning stage, where θ = {θ base , θ novel }, θ base is the trained base classifier and θ novel is the novel classifier randomly initialized. Also, in order to make all the classes well balanced in the fine-tuned model, existing GFSL methods (Kukleva, Kuehne, and Schiele 2021; Qi, Brown, and Lowe 2018) train a balanced dataset, which is either a subset or superset of D base ∪ D novel , constructed by undersampling D base or oversampling D novel . Given such a balanced dataset, most state-of-the-art methods (Ren et al. 2019;Yoon et al. 2020) fine-tune only the classifier while freezing the feature extractor. This is due to the fact that the feature extractor is already well-trained on a sufficient number of samples and hence better to be frozen not to hurt generalization of the model.
As long as a balanced dataset is ideally constructed, the model can well be trained for both base and novel classes by this basic framework as fine-tuning can properly rearrange all decision boundaries in the joint feature space. Unfortunately, vanilla sampling strategies (Huang et al. 2016;Wang, Ramanan, and Hebert 2017) would not work well in GFSL as undersampling inevitably causes information loss and oversampling inherently suffers from the lack of diversity due to many redundant samples. This is why many existing works propose extra techniques to achieve a better performance, which is not quite satisfactory according to our experiments. More importantly, a balanced dataset can be collected and fine-tuned only if the base dataset is available, which may not always be the case.

Problem Statement of Zero-Base GFSL
In the proposed zero-base GFSL problem, our objective is to train a joint linear classifier θ, which works well for both C base and C novel , with only a few samples of novel classes (i.e., D novel ) without using any samples of D base . Following the basic GFSL framework, we only fine-tune the classifier while freezing the feature extractor as: The weight distributions of base classifier The weight distributions of novel classifier Note that it is even more desired to freeze the feature extractor in zero-base GFSL than in GFSL because the feature extractor can severely be biased toward novel classes after fine-tuning with only novel samples.

Analysis and Methodology on Zero-Base GFSL
This section first investigates why the basic framework of GFSL cannot perform well in zero-base GFSL, and then presents an effective weight normalization method that enables a simple transfer learning scheme to achieve even a better performance than existing GFSL methods assuming the existence of base data.

Hardness of Zero-Base GFSL
As presented in Figure 2, a model trained by the basic framework of GFSL, that is, fine-tuning only the joint classifier θ with D novel , turns out to be extremely inaccurate for base classes yet pretty accurate for novel classes despite freezing the feature extractor. Thus, the fine-tuned classifier gets highly biased toward novel classes due to the absence of base samples. More specifically, when freezing the feature extractor φ, the feature space itself is fixed and so are the feature vectors f φ (x) of all the samples x. Consequently, it is decision boundaries in the space that are changed by fine-tuning the classifier and therefore matter to a biased prediction. Without a balanced dataset, it is challenging to properly learn hidden relationships between base and novel classes, leading to enlarged (i.e., biased) decision boundaries of novel classes as depicted in Figure 1. In zero-base GFSL, it is our mission to find the best balance between decision boundaries of base and novel classes without retraining any base samples so that we can ideally make the best joint prediction. As decision boundaries between base and novel classes are formed where θ base f φ (x) = θ novel f φ (x), they are mainly affected by how the weight values of θ base and θ novel are distributed. In the following subsection, we therefore conduct a systematic analysis on the weight distributions of θ base and θ novel .

Analysis on Weight Distributions of Classifiers
We investigate the underlying distributions of weights of base and novel classifiers particularly in terms of their mean µ and variance σ 2 . More specifically, given a classifier θ = [θ 1 , θ 2 , . . . , θ |C| ] ∈ R d×|C| , where θ i ∈ R d represents the weight vector corresponding to class i, we first define the vectors of class-wise means and standard deviations of weight values as follows: Also, we use µ base , σ base , µ novel , and σ novel to denote the vectors corresponding to the weights of base and novel classifiers (i.e., θ base and θ novel ).
In order to examine how the weight values of the novel classifier are different from those of the base classifier, we compute the average of class-wise means and standard devi- σ i and compareμ novel andσ novel withμ base andσ base . As observed in Figure 3, we discover the following two undesirable facts, both of which should be tackled to obtain a balanced joint classifier. L2-norms proportional to standard deviations. We first discover that the variance of θ base is greater than that of θ novel , i.e.,σ base >σ novel , as shown in Figure 3. This is somewhat intuitive in that the base classifier is trained on a large number of samples in D base that can increase the diversity, whereas the novel classifier cannot have such a large diversity due to the lack of training samples in D novel . What is not quite obvious is that these standard deviations are indeed proportional to the L2-norms of weights in neural networks as the following proposition. Proposition 1. Consider a parameter θ of N weights in a neural network with the assumption that θ is randomly initialized by a Gaussian distribution with zero mean and trained by a uniformly distributed dataset. Then, it holds that By Proposition 1, θ base 2 > θ novel 2 is implied from the observation ofσ base >σ novel . Somewhat surprisingly, this contradicts to the highly biased results toward novel classes in Figure 2 in the sense that many existing works (Hou et al. 2019;Kang et al. 2020;Zhao et al. 2020) report that larger weight norms of a class tend to produce larger logits leading to more biased predictions toward the corresponding class. Thus, if we follow the existing strategy of equalizing the weight norms (Hou et al. 2019;Kang et al. 2020;Zhao et al. 2020) by increasing the smaller ones yet decreasing the larger ones, the novel-biased classifier in zerobase GFSL would get even more biased toward novel classes, to be confirmed in our experiments.
Then, how could the classifier be highly biased toward novel classes even though the weight norms of novel classes are less than those of base classes? The true answer to this question lies in the average weight of each classifier rather than its weight norm. Mean shifting phenomenon. In terms of means of weight distributions, we observe the fact that the average weight of the novel classifier is positively shifted (i.e.,μ novel > 0) while that of the base classifier keeps almost zero (i.e., µ base ≈ 0). We call this situation mean shifting phenomenon commonly observed in all the graphs of Figure 3.
Positively shiftedμ novel indicates that each weight of θ novel on the average has a more positive value. Recalling that decision boundaries are constructed at θ base f φ (x) = θ novel f φ (x), more positive weights of θ novel would increase θ novel f φ (x), and therefore enlarge the decision boundaries of novel classes. This implies that mean shifting phenomenon is a more critical reason behind the novel-biased model in zero-base GFSL. As mentioned above, however, the existing works (Hou et al. 2019;Kang et al. 2020;Zhao et al. 2020) only focus on equalizing the weight norms, which turn out to be proportional to the standard deviations of weights, between base and novel classes, and hence their normalization method alone fails to achieve a satisfactory performance. What makes mean shifted. Let us now analyze why mean shifting phenomenon occurs to the novel classifier in zerobase GFSL. To this end, we first consider how we update the weights of each classifier by gradient descent as θ = θ − η ∂L ∂θ (f φ (x), θ), where the gradient is: Since f φ (x) is always non-negative due to the ReLU function and p(x) is also a non-negative probability value, the gradient is positive only if y i = 1 (i.e., for the probability of the class label of x) and negative for all the other class probabilities.
In training a balanced dataset of all the classes uniformly distributed, the total positive gradient for each class eventually gets similar to the absolute value of its total negative gradient, which is whyμ ≈ 0. Thus, if we train D novel only to θ novel , µ novel would also be close to zero. However, when fine-tuning the joint classifier θ = {θ base , θ novel } with only D novel , the total positive gradient of θ novel increases whereas the absolute value of its total negative gradient decreases because all the output probabilities of θ novel together become smaller in training θ than they used to be in training only θ novel . To illustrate, consider an example of Figure 4, where output probabilities of θ novel are larger than those of θ = {θ base , θ novel } for a given training image of 'cat'. Thus, even if θ novel outputs [0.7 0.3] for the cat image, the joint classifier θ would output something like [0.05 0.05 0.04 0.06 0.05 0.05 | 0.55 0.15] for 6 base classes followed by 2 novel classes. Consequently, the positive gradient caused by the novel class 'cat' is increased from 0.3 to 0.45, but the absolute value of the negative gradient for the second novel class 'bird' is decreased from | − 0.3| to | − 0.15|. Furthermore, θ novel can receive negative gradients only from the instances of the other novel classes due to the absence of base instances. This makes µ novel to be more positively shifted.
As for θ base , only negative gradients will arrive during the entire fine-tuning process, but its total amount would not be that much because the well-trained θ base is not likely to be overconfident to irrelevant novel classes particularly when θ novel gets used to identifying their corresponding novel classes. This is why µ base still keeps to be zero after finetuning.

Solution: Mean-Variance Classifier Normalization
To fulfill both the zero-mean and a balanced variance of base and novel classifiers, we propose a simple yet effective normalization method, called Mean-Variance Classifier Normalization (MVCN). MVCN takes two essential steps, namely online mean centering and offline variance balancing. We perform online mean centering during fine-tuning the joint classifier, and then proceed to balance variances of weights as a post processing step once the classifier is well trained. The entire process is outlined in Figure 5. Online mean centering. First, we keep normalizing the weights of the novel classifier to have zero-mean in the process of training by: When fine-tuning the joint classifier, online mean centering is performed only on the novel classifier. This directly reduces the positive weights of the novel classifier and keeps the zeromean while learning the features of novel classes. Note that we do not give any constraints to the variance of weights during fine-tuning, but rather allow each classifier to learn as many required features as possible.
Offline variance balancing. Once the fine-tuning stage is done, we adjust the weights of base classifier according to the ratio of standard deviations as follows: whereσ novel is the average of class-wise standard deviations for all novel classes. We re-scale each weight of the base classifier by multiplying the ratio of the standard deviation of the novel classifier to that of the base classifier (i.e.,σ novel σ base ), and thereby the weight variance of the base classifier gets similar to that of the novel classifier. Note that we do not directly normalize θ by their σ, which makes the zero-mean and unit-variance θ−µ σ , because the standard deviation of weights is much smaller than 1. Post linear optimization without base data. Finally, we introduce class-wise learnable parameters γ i and β i for i ∈ C base ∪ C novel , which are intended for further optimizing decision boundaries of novel classes. With freezing θ = {θ base , θ novel }, γ i and β i are similarly trained by Eq.  Figure 5: Overview of Mean-Variance Classifier Normalization. In the pre-training stage, we train the entire model on all base classes. In the fine-tuning stage, we normalize the novel classifier by online mean centering in the process of training and adjust the trained weights of the base and novel classifiers by re-scaling them using the standard deviation ratio.

Mismatch of base and novel weight distributions
(2) and used at inference time as γ · θ f φ (x) + β. Through these additional parameters, we can further improve the performance of novel classes particularly in an extreme case of 1-shot learning. Note that this optimization process is performed still without using any base samples.

Experiments Experimental Settings
Datasets. We compare our method with the state-of-the-art (SOTA) GFSL methods using two datasets, mini-ImageNet (Vinyals et al. 2016) and tiered-ImageNet (Ren et al. 2018), which are most widely used in the literature of GFSL. The mini-ImageNet contains 100 classes and 60,000 sample images from ImageNet (Russakovsky et al. 2015), which are then randomly split into 64 training classes, 16 validation classes, and 20 testing classes, proposed by (Ravi and Larochelle 2017). The tiered-ImageNet is another subset of ImageNet, containing 608 classes, which are then split into 351 training, 97 validation, 160 testing classes. This setting is more challenging since base classes and novel classes come from different super classes. The size of all images in both datasets is 84 × 84. In addition, we test whether our method works better than the SOTA GZSL (Generalized Zero-Shot Learning) methods using the three most common datasets, CUB (Welinder et al. 2010), AWA1 (Lampert, Nickisch, and Harmeling 2009), and AWA2 (Xian et al. 2019). Implementation details. We implement all the methods in PyTorch, and train each model on a machine with NVIDIA A100. We use ResNet-12 (He et al. 2016) for mini-ImageNet and ResNet-18 for tiered-ImageNet, according to (Ren et al. 2019;Yoon et al. 2020). For CUB, AWA1 and AWA2, we commonly use ResNet-101. Full details of our settings are covered in Appendix.

Experimental Results
Overall performance. Table 1 summarizes the overall performance of the compared GFSL methods on ImageNet. Even without base data, it is clearly observed that our MVCN method outperforms the other GFSL methods exploiting base samples. In tiered-ImageNet, which is a more challenging scenario where base and novel classes are quite different, MVCN seems to be even more effective when it is in mini-ImageNet. This is probably because a larger number of base classes can be pretrained in tiered-ImageNet than in mini-ImageNet, and the simple transfer learning scheme of MVCN takes more advantage of this well-trained knowledge of many base classes. Especially when we have more novel samples like 5 or 10-shot, the performance gap between ours and the other methods becomes larger as shown in Table 1. The underlying reason is mean shifting phenomenon gets stronger as we finetune more novel classes (see Figure 3). Although XtarNet (Yoon et al. 2020) occasionally performs slightly better on base classes than ours, MVCN still manages to outperform XtraNet on novel classes with clear margins. Furthermore, Table 2 shows that MVCN is superior to all the SOTA GZSL methods on CUB, AWA1, and AWA2. Note that each image in these datasets is augmented with textual attributes that are crucially utilized by most GZSL methods while our method does not exploit any extra information. Qualitative results. The confusion matrices of with and without normalization are presented in Figure 6. We can  Table 1: Performance (%) comparison with GFSL methods on mini-ImageNet (top) and tiered-ImageNet (bottom).
Methods/ Shots 1-shot 2-shot 5-shot 10-shot 1-shot 2-shot 5-shot 10-shot 1-shot 2-shot 5-shot 10-shot   Table 3: Ablation study on mini-ImageNet to analyze the effectiveness of three components of our method, which are mean centering (MC), variance balancing (VB), and linear optimization (LO). see that the linear classifier without normalization tends to erroneously classify base samples as novel classes, but not vice versa. This observation confirms our claim that decision boundaries of novel classes get unnecessarily large in the finetuned classifier. However, after applying our normalization scheme, it is well observed that the classifier seems no longer to be confused about base classes with novel classes.
In addition, as shown in Figure 7, we can observe that MVCN actually resolves mean shifting phenomenon and achieves a balanced variance between base and novel classes as the classifier with MVCN has aμ novel 62 times smaller than it is without normalization. Also, we show thatσ base and σ novel become almost the same after normalization, implying that there is no bias toward either novel or base classes. Ablation study. To examine the effect of each component of our method, which are online mean centering (MC), offline variance balancing (VB), and post linear optimization (LO), we conduct an ablation study in zero-base GFSL using mini-ImageNet. Table 3 shows that applying mean centering solely improves the accuracy of base classes by more than 25% in all shot cases, which shows its effectiveness of mitigating the novel-bias problem. In addition to MC, variance balancing tends to make the model more accurate for novel classes yet a bit less accurate for base classes. Considering the results of Table 1, note that MC together with VB beats the SOTA GFSL method XtarNet by large margins in 5-shot case, and shows comparable performance to the best 1-shot learning method LCwoF. Finally, when optimizing linear parameters with a tiny overhead, we can further improve the performance with 1.46% higher accuracy for the 1-shot case. Through all the components, we get 4.96% and 12.01% performance gains for 1-shot and 5-shot, respectively.

Conclusion
In this paper, we conducted the first study on zero-base GFSL, where we need to fine-tune a joint classifier with only a few samples of novel classes. Through a systematic analysis, we discovered that mean shifting phenomenon was the critical reason behind a novel-biased classifier, but the existing GFSL methods have been trying to equalize only the variance of weights. Based on our findings, we proposed a simple yet effective weight normalization method without using any base samples, which can even beat the existing GFSL methods that utilize the base dataset. Even though this work dealt with only linear classifiers, which is our limitation, our next plan is to extend our analysis to cover various types of classifiers. Table A1: Ablation study on ImageNet-800 to analyze the effectiveness of different components in our method. MC is the mean centering, VB is the variance balancing, and LO is the re-scaling with additional linear parameters.  Table A2: Performance comparison of various naïve approaches using mini-ImageNet.
the same as those of the original ImageNet dataset. Note that this large dataset has never been tested by any of the existing GFSL methods.

Additional Experiments
Ablation Study Table A1 presents the detailed results of another ablation study using ImageNet-800, whose summary is shown in Figure 2. As mentioned in the introduction section, our normalization method almost reaches to the upper bounds of base and novel classes particularly when we are able to use a powerful pretrained model such as ResNet-50 trained from abundant data ImageNet-800. The upper bound of novel classes is referred by a previous work (?) of measuring only the novel classes, and the upper bound of base classes is taken by the original performance of the pretrained model.

Comparison with Naïve Approaches
In this section, we compare our method with a variety of naïve approaches, which are not necessarily introduced by the existing works. Specifically, we consider the following six methods: • Linear classifier Fine-tuning only the default linear classifier • Freezing base classifier Fine-tuning only the novel classifier while freezing the base classifier • Cosine classifier Fine-tuning the cosine classifier, instead of a linear classifier, often used by the existing GFSL methods (Gidaris and Komodakis 2018; Qi, Brown, and Lowe 2018) • L1 regularization L1 regularization for mitigating mean shifting phenomenon, instead of online mean centering • L2 regularization L2 regularization for mitigating mean shifting phenomenon, instead of online mean centering • VB in training Variance balancing during fine-tuning while the others are the same as our normalization method   Table A4: Comparison of different feature extractors on tiered-ImageNet.
In Table A2, we show that our MVCN method takes always the best position among all the compared approaches. Note that even though we freeze the base classifier, the resulting accuracy of base classes does not remain the same, but abruptly decreases. This is because, as mentioned earlier, mean shifting phenomenon harms the overall balance between novel and base classifiers, which is not addressed by freezing the base classifier. Given the occurrence of mean shifting phenomenon, using the cosine classifier seems to make the situation even worse as observed that the accuracy of base classes gets lower than it is when using the linear classifier. This is due to the fact that the cosine classifier is effective to equalize the weight variances between novel classes and base classes by increasing the weight norms of the novel classifier but decreasing those of the base classifier. Consequently, the novel bias problem gets more severe when using the cosine classifier without addressing mean shifting phenomenon. Strong L1 or L2 regularization improves the accuracy of base classes, implying its ability to mitigate mean shifting phenomenon. However, this time the model gets significantly in accurate for novel classes, which is because L1 or L2 regularization tends to reduce the weight variance (i.e., weight norm) of novel classes. Thus, these regularization approaches cannot achieve a desirably balanced result between novel and base classes. Finally, we can see that variance balancing is much better to be performed as a post-training process in offline, rather than being performed with mean centering in the process of training.

Mean Centering for Both vs. Novel Only
We investigate how the experimental results can be changed if we apply mean centering not only to the novel classifier but also to the base classifier. As shown in Table A3, when conducting mean centering for both base and novel classifiers, the performance of base classes increases by more than 5% as negative mean shifting (e.g.,σ base = −0.0029 in Figure 3) of base classes gets addressed as well. When it comes to the overall performance, however, it is still observed that mean centering only for the novel classifier is better than applying it to both classifiers. Thus, we can control the balance between base and novel classes by adjusting to what extent mean centering is performed to either the base or novel classifier. Table A3 also shows how much the final accuracy gets improved if we additionally use base data just like the normal GFSL scenario. Although there is a slight improvement in accuracy over GFSL (i.e., with base data) to zero-base GFSL, the performance gap is surprisingly subtle. We believe that our normalization method successfully plays the role of base data to such an extent that additional fine-tuning with a balanced dataset, which is obtained by undersampling, cannot make much difference.

Utilization of More Powerful Feature Extractors
It has been reported that the better feature extractors (i.e., often deeper architectures) we use, the better the performance we can achieve in standard FSL (Chen et al. 2019;Dhillon et al. 2020;Tian et al. 2020). Unfortunately, this has not been the case in GFSL as the existing GFSL methods (Gidaris and Komodakis 2018;Shi et al. 2020) do not seem to effectively leverage of such a deeper architecture. In order to examine how well our CN method utilizes the power of feature extractors, we conduct the experiments using two different feature extractors, namely ResNet-12 and ResNet-18, both of which are trained on Tiered-ImageNet. As shown in Table A4, for 1-shot and 5-shot, CN with ResNet-18 leads to 2% and 2.7% improvements in accuracy, and 3.5% performance gain particularly for novel classes 1 . This indicates that our CN method works better when it is with a better (i.e., stronger) feature extractor, which has been as nicely observed in standard FSL.

Effect of Linear Parameters
Finally, we analyze the influence of γ and β as shown in Figure A1. The ∅ iteration means the performance before variance balancing, and 0 iteration means the performance right after performing variance balancing. When it is hard for novel classes