{'fraction_non_alphanumeric': 0.042091752780885935, 'fraction_numerical': 0.015072730862897387, 'mean_word_length': 4.586872586872587, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 2, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Worst Case Matters for Few-Shot Recognition

Few-shot recognition learns a recognition model with very few (e.g., 1 or 5) images per category, and current few-shot learning methods focus on improving the average accuracy over many episodes. We argue that in real-world applications we may often only try one episode instead of many, and hence maximizing the worst-case accuracy is more important than maximizing the average accuracy. We empirically show that a high average accuracy not necessarily means a high worst-case accuracy. Since this objective is not accessible, we propose to reduce the standard deviation and increase the average accuracy simultaneously. In turn, we devise two strategies from the bias-variance tradeoff perspective to implicitly reach this goal: a simple yet effective stability regularization (SR) loss together with model ensemble to reduce variance during fine-tuning, and an adaptability calibration mechanism to reduce the bias. Extensive experiments on benchmark datasets demonstrate the effectiveness of the proposed strategies, which outperforms current state-of-the-art methods with a significant margin in terms of not only average, but also worst-case accuracy. Our code is available at https://github.com/heekhero/ACSR.


Introduction
Most people have the ability to learn to recognize new patterns via one or a few samples (e.g., images), thanks to the accumulated knowledge. Naturally, fewshot learning [35] aims at learning from scarce data, which is already studied long before the deep learning era. In this paper, we focus on the image recognition task, also known as few-shot image classification, a widely studied few-shot task [33,29,9,30,5,21,1,23,7,2,36]. Deep learning techniques have further pushed few-shot learning's average accuracy over multiple runs (i.e., episodes) towards a high level that appears to be already applicable to real-world applications.
In this task, a pretrained model is first derived from the base set, a large set of labeled images. Then, given some unseen categories and very few training images per category (the novel set), the model must learn to adapt to classifying new examples from these novel categories. Differently sampled novel sets lead to different episodes, different trained models and test accuracies. The common Probability Mass Accuracy (%) 3 3 evaluation criterion is to run a large number of (usually 500 to 10000) episodes for the same task, and report the average accuracy and its 95% confidence interval. We aim at making few-shot recognition more practical, too. But, we argue that both metrics (mean accuracy and 95% confidence interval) are not helping us towards reaching this goal, given the recent progress in this task. Fig. 1 shows the distribution of accuracy of 500 episodes for the same task, whose average accuracy estimate is 68.96% and the 95% confidence interval is [68.07, 69.85]this interval has 95% chance of the true average accuracy landing on it, not the chance of a single episode's accuracy dropping inside! In other words, both metrics are used to describe the mean accuracy of 500 episodes, not the accuracy of a single experiment.
In fact, the worst episode among these 500 in Fig. 1 has only 37.33% accuracy while that of the best is close to 100%. That is, few-shot learning is very unstable, and the accuracy varies dramatically. The worst-case lags far behind the average.
Furthermore, now that we aim at making few-shot learning practical, we have to accept that in most real-world applications, we can run the experiment only once, but do not have the luxury of running 500 (or more) experiments and pick the best or average episode among them. In other words, the worst-case scenario (if one is unlucky) is naturally more important than the average casein commercial applications, we welcome a product with an acceptable worst-case performance more than one that performs well on average but occasionally goes calamitous. We argue that we need to pay attention to one episode instead of the average of 500 episodes, and to maximize the worst case's accuracy instead of the mean.
But, it is in general very challenging to explicitly optimize the worst-case scenario even when there are many training examples, let alone in the few-shot setting. We propose an alternative, indirect line of attack. As shown in Fig. 1, the accuracy distribution often fits well to a Gaussian. Then, the worst-case accuracy is naturally estimated by the 3σ rule as µ − 3σ, where µ is the average accuracy and σ is the accuracy's standard deviation-in a normal distribution, the chance to fall on the left of µ − 3σ is only 0.135%. Fig. 1 clearly supports this approximation. In other words, we do need to increase the mean accuracy µ (as in current methods), but also need to simultaneously minimize the standard deviation σ.
Unfortunately, directly optimizing σ is not plausible, because it involves different episodes and different models, which is beyond the capability of end-to-end deep learning. To tackle this issue, we propose to resort to the bias-variance decomposition theory [14], which states that the expected error (1 − µ) is the sum of squared bias plus variance (σ 2 ). From this classic theory, we know that stable learning machines lead to small σ, and hence propose a novel stability regularization (SR) to utilize the base set to achieve stability. We also incorporate model ensemble, because it is well-known for its effect of reducing both bias and variance. Similarly, this theory also states that a model with larger capacity often lead to smaller bias, hence we propose an adaptability calibration (AC) strategy to properly increase the model capacity without overfitting, so as to indirectly reduce the bias and to increase the average accuracy µ in turn.
Our contributions can be summarized as follows: 1. To the best of our knowledge, we are the first to emphasize the importance and to advocate the adoption of worst case accuracy in few-shot learning. 2. Motivated by Fig. 1, we argue that in addition to maximizing the average accuracy µ, we must also simultaneously reduce the standard deviation σ. 3. We propose to achieve this goal from the bias-variance tradeoff perspective.
We propose a simple yet effective stability regularization (SR) loss together with model ensemble to reduce variance during fine-tuning. The SR loss is computed in an unsupervised fashion, and can be easily generalized to data beyond the base set. We also propose adaptability calibration (AC) to vary the number of learnable parameters to reduce bias.
As a result, our method not only achieves higher average accuracy than current state-of-the-art methods, but more importantly enjoys significantly higher worst case accuracy on benchmark few-shot recognition datasets.

Related Work
Generalizing from base categories, few-shot learning aims at designing effective recognition systems with limited training data from novel categories. Current few-shot learning research can be roughly divided into two branches: those based on meta learning and those based on transfer learning. We now give a brief introduction of representative methods within both branches. Besides, since we propose using the ensemble technique for better regularizing the model stability, model ensemble methods in few-shot problems will be discussed as well.
Meta learning. This line of methods model the training process of few-shot learning by pretending the episodes (which are usually large in quantity) are to be learned in a way of "learning to learn". One line of work [9,20,22,28,10] learns a set of good initial weights to fast adapt to unseen episodes with a limited number of gradient descent steps. Another line of work [29,30,33,4,17] leverages the characteristics of different distance metrics to classify unknown samples by comparing with embeddings or their variants (e.g., prototypes) derived from the training examples.
Transfer learning. These methods [5,13,3,12,8,21,1,23,7,2,16,36] take advantage of the standard transfer learning pipeline, which first pretrain a model on base classes, then revise the feature embeddings output by the pretrained model with limited novel samples. [5] used cosine classifier to normalize the magnitude of both embeddings and classification weights for compacting the intra-class intensity. [12] proposed to pretrain with self-supervised auxiliary tasks for boosting few-shot learning. [21] introduced a negative margin loss during pretraining to increase the discriminability of features. [1] used the associative alignment strategy to learn novel features by aligning them with closely related samples from the base set. [23] explored the way to pretrain with manifold mixup [32] for good generalization. [7] proposed a baseline for transductive fine-tuning with novel samples. [16] introduced a new pipeline, which first preprocessed features, then classified them with an optimal-transport inspired algorithm. [36] adjusted novel distributions using base categories to generate imaginary training samples.
Model ensemble. It is a well-known strategy [14] to incorporate a number of weak models to build a strong model, which will effectively increase the stability and robustness during inference. Very recently, for the few-shot classification problem, [8] harmonized the cooperation and diversity between deep models to pursue better ensemble performance. E 3 BM [22] tackled few-shot problems with the ensemble of epoch-wise base-learners whose hyper-parameters were generated by task-specific information. Different from these methods, our ensemble stability regularization directly increases the diversity of deep models by dealing with different parts of data from the base set, keeping simplicity while achieving superior performance.

The Worst-case Accuracy and Its surrogate
We first give some background information about how a few-shot problem is defined, then describe the relationship between the commonly used 95% confidence interval and the standard deviation (σ). Lastly, our solution that advocates indirectly optimizing for the worst-case will be presented.
In the few-shot recognition setting, there exists a dataset with abundant labeled images called the base set, denoted as is its corresponding category label, and N b is the number of examples. In addition, there also exists another dataset with scarce labeled images from new categories (usually 1 or 5 per category) called the novel set, denoted as D n = {x n j , y n j } Nn j=1 , where y n j ∈ Y n and Y b ∩ Y n = ∅, with N n being the number of examples it contains. In every episode, N categories will be sampled to constitute the novel set, and K examples will be sampled Table 1. 5-way 1-shot recognition results of existing methods on mini-ImageNet (500 episodes). ♭ means that σ is calculated by our implementation due to undisclosed n in published papers. ACCm, ACC1 and ACC10: higher is better; Z 95% and σ: lower is better. The best results are shown in boldface.

Method
ACCm Z 95% σ ACC1 ACC10 from each of those categories (called N -way K-shot). Learned with this tiny set of images (and optionally D b ), a few-shot recognizer needs to recognize unseen examples from the N categories in this episode.

Existing and Proposed Metrics
Existing methods evaluate their performance by averaging the accuracy of n episodes, including the average accuracy µ and a 95% confidence interval [µ − Z 95% , µ + Z 95% ]. As aforementioned, both metrics are estimates for the average accuracy random variable, not estimating the accuracy of one episode. Hence, the interval radius Z 95% is often surprisingly small, as shown in Table 1.
As established in Sec. 1, we need to focus more on the worst-case accuracy among all n episodes, which is denoted as ACC 1 . In addition, we also report the average accuracy of the 10 worst cases as ACC 10 . The empirical average accuracy is denoted as ACC m . Although it is a general trend that ACC m and ACC 1 are positively correlated, the 6 methods rank significantly differently using ACC m and ACC 1 . For example, S2M2 R is 3.64% lower than LR-DC in terms of µ (ACC m ), but 0.25% higher in ACC 1 (worst-case accuracy). That is, although maximizing µ is useful, it is far from being enough. We argue that for few-shot recognition to be practically usable, we need to maximize ACC 1 instead.
The Z 95% metric is also misleading, as it measures uncertainty in estimating µ. In fact, based on its definition, we have where σ is the standard deviation of accuracy across episodes. Hence, a small Z 95% may well be because n is large (e.g., n = 10000), instead of due to a small σ, which is illustrated clearly in Table 1. For example, MixtFSL has almost 3x larger Z 95% than that of CGCS, but has a smaller σ within the pair. And different papers often use different n values, which renders Z 95% difficult to interpret. Furthermore, the proposed worst-case accuracy ACC 1 is not only semantically more meaningful than ACC m (µ), but also more stable. We use ACC k to denote the average accuracy of the k worst episodes. Results in Table 1 exhibits that ACC 1 and ACC 10 ranks the methods consistently.

Implicitly Optimizing the Worst-Case Accuracy
As briefly introduced in Sec. 1, we can only implicitly maximize ACC 1 , and we propose to use µ − 3σ as a surrogate. As empirically shown in Fig. 1, when the accuracy distributes as a Gaussian, µ − 3σ is a perfect surrogate, as it pins to the 0.135-th percentile of ACC 1 , because Φ(−3) = 0.00135 in which Φ is the cumulative distribution function of the standard normal distribution N (0, 1). Even if the accuracy distribution is highly non-normal (which empirical data suggests otherwise), the one-sided Chebyshev inequality also guarantees that µ − 3σ is no larger than the 10th percentile (i.e., no better than the 10% worst cases), because the inequality states that Pr(X ≤ µ − kσ) ≤ 1 1+k 2 for any distribution X and any k > 0, while k = 3 in our case.
But, this surrogate loss is still a qualitative one instead of a variable that can be directly maximized, because σ non-linearly involves all n episodes, including the n models and the n training sets. Hence, we transform our objective to simultaneously maximize µ and minimize σ, and propose to use the bias-variance tradeoff for achieving both objectives indirectly.

The Bias-Variance Tradeoff in the Few-Shot Scenario
The bias-variance decomposition states that the expectation of error rate (i.e., 1 − µ) equals the sum of the squared bias and the variance [14]. 1 Although the definitions for bias and variance of a classifier are not unique, the following qualitative properties are commonly agreed upon [14]: 1. A classifier has large variance if it is unstable (small changes in the input cause large changes in the prediction), and vice versa; Hence, we expect a smaller σ (equivalently, variance) if we make the recognizer more stable. 2. A classifier with larger capacity (e.g., more learnable weights) in general has a smaller bias, and vice versa. Hence, we prefer a model with larger capacity to reduce the bias. 3. Although minimizing both bias and variance simultaneously amounts to maximizing the expected accuracy, the two terms are often contradictory to each other. Larger models are more prone to overfitting and in turn larger variance, too. Hence, a properly calibrated capacity increase is necessary to strike a balance.

Reducing Variance: Stability Regularization
To fulfill these goals, our framework (cf. Fig. 2) follows the common practice to train a backbone network f (x) (its classification head discarded) using the base set D b . Then, in one episode, a N -way K-shot training set is sampled as the novel set D n to fine-tune the backbone (with a randomly initialized classification head W ) intof (x) using a usual cross entropy classification loss L C .  Fig. 2. The proposed few-shot recognition framework. A backbone network f (x) is trained using the base set samples and its classification head is discarded. Using images from the novel set, the backbone is fine-tuned intof (x), plus a new classification head W . Stability off (x) is maintained by the stability regularization (SR) loss LS, while the capacity increase (adaptability) is calibrated by fine-tuning a selected subset of layers in the backbonef (x). The classification loss LC minimizes error (i.e., to increase the average accuracy µ). Best viewed in color.
The danger is: because K is very small (mostly 1 or 5), the fine-tuning process easily gets overfit and is extremely unstable-different sampled training sets lead to dramatically different prediction accuracy. To increase the stability (and thus to reduce the variance σ 2 ), we must not allow the weights inf (x) to be dominated by the small novel training set and completely forget the representation learned from the base set, i.e., the knowledge in f (x).
Because the base classification head has been discarded, the stability is regularized by requiring the original learned representation (f (x) for an input x) and the fine-tuned representation (f (x) for the same input) to remain similar to each other, or at least not to deviate dramatically. A simple negative cosine similarity loss function realizes this stability requirement: In order to avoid overfitting, the x in our stability regularization (SR) loss is not sampled from the novel set. In each mini-batch, we randomly sample 256 images from the base set with replacement to calculate L S and back-propagate tof (x) (but f (x) is frozen). The proposed SR loss is minimized iff (x) and f (x) are the same (modulo a scale factor), which makesf (x) produce similar representations to f (x) for all base set images, hence we can expect that it will be stable given different training sets sampled from the novel split. It is worth noting that the proposed SR loss is very flexible since only unlabeled images are required to compute the loss. Hence, it can be easily extended to use other unlabeled images. Results on using images from other than the base or novel set are presented in Sec. 4.4. The other loss function L C in Fig. 2 is a regular cross entropy loss, which aims at maximizing the average accuracy µ. Hence, the overall objective is L = L C + α · L S , where α is always 0.1 in all experiments.

Reducing Bias: Adaptability Calibration
As aforementioned, to reduce the bias, we adjust the capacity of our model. In many few-shot learning methods [21,2,36,29,16], either the features extracted by the backbone f (x) are directly used, or the backbone is freezed during finetuning. In other words,f (x) ≡ f (x) and the capacity of the model is only determined by the linear classification head W (or its alike), which has very low capacity, and in turn leads to high bias according to the bias-variance tradeoff. Other methods [7] fine-tune the entire backbone network, i.e., all parameters in the backbone f (x) are learnable andf (x) is completely different from f (x). Such a high capacity model inevitably leads to overfitting and ultra high variance, despite having a small bias. As aforementioned, we need to strike a good balance.
We propose a simple remedy called adaptability calibration (AC), which freezes part of the layers in the backbone f (x) and fine-tune the rest layers to formf (x). The less layers frozen, the higher is the model's adaptability. The calibration is experimentally determined.
Backbone models in the ResNet [15] family have 5 groups of residual blocks, denoted as 'res1' (close to the input) till 'res5' (close to the classification head W ). In Fig. 3, from left to right we gradually make 'res5' to 'res1' learnable. In the 5-shot scenario, allowing both W and 'res5' updatable in fact consistently strikes the best tradeoff between bias and variance. However, in the difficult 1-shot case, only learning W seems to be the best while W +'res5' is the runnerup. Hence, considering both cases, our adaptability calibration chooses to update both W and 'res5' (the column '5' in Fig. 3).

Reducing both Variance and Bias: Model Ensemble
It is well-known that model ensemble is an effective way in reducing both the variance and the bias of a classifier system [38], and hence it is potentially useful for our task. Diversity among models in the ensemble (i.e., models need to predict differently for the same input) is crucial to the success of ensemble learning, while diversity is often obtained by sampling different training sets (or example weights) for different models on the same task (e.g., in one episode of few-shot learning).
However, the sampled novel sets for fine-tuning few-shot recognizers are so small that it leaves almost no space for sampling training sets once more for ensemble learning (e.g., consider a 1-shot task). With the proposed stability regularization and its inclusion of the base set D b , ensemble learning becomes possible and natural.
We randomly divide the base set D b into M = 4 disjoint subsets, D m b (m = 1, 2, 3, 4). Then we train M few-shot recognizers for one episode, each trained with the sampled novel set and stability regularized by one D m b . During inference, the classification probabilities output by these M models are averaged to generate recognition results of the ensemble model.
Training Details. To validate the effectiveness of the proposed method, we take WRN-28-10 [37] or models in the ResNet series [15] as the backbone f (x), and the cosine classifier used in [5] as the classification head W . We follow [23] to conduct pretraining. At the fine-tuning stage, we trainf (x) and W by adapting the pretrained model f (x) to novel categories. For each few-shot episode, the sampled novel set is used to fine-tune the model for 100 epochs with the label smoothed cross-entropy loss [31] (ϵ=0.1), by using SGD with learning rate 0.1, weight decay 1e-4 and momentum 0.9.
Evaluation Protocols. We conduct evaluation in the form of N -way Kshot Q-query, where firstly N categories from the novel split of different datasets are selected arbitrarily, then mutually exclusive K and Q samples of each category are further chosen in the same manner, respectively. The N × K samples Table 2. 5-way accuracy (%) on mini-ImageNet, CUB and CIFAR-FS with WRN-28-10 as the backbone. ACCm is the average accuracy copied from published papers if not otherwise noted.

mini-ImageNet
ProtoNet † [29] 54. 16  ♢ ACCm from our implementation † ACCm from [5] are regarded as D n to adapt the pretrained model to these new categories, afterwards the model is evaluated on the N × Q test samples. The above process is collectively called one episode. We keep N =5, Q=15 and vary K=1 or 5 to conduct experiments. We report the average of all (Acc m ), worst 1 (Acc 1 ), average of worst 10 (Acc 10 ), average of worst 100 (Acc 100 ) accuracy and standard deviation (σ) over 500 pre-sampled episodes by 5 runs then average. Specifically for previous methods with only Acc m reported, we get other results following their official implementations. All experiments we conduct on one dataset are based on the same 500 episodes, which makes the comparison fair.

Comparing with State-of-the-art Methods
We compare 5-way accuracy of our method with state-of-the-art methods in Table 2. 'AC' and 'SR' stand for adaptability calibration and stability regularization, respectively, and 'EnSR' means ensemble of SR.
The proposed method (denoted as AC+SR) outperforms existing methods using the traditional ACC m metric by 1-5% in both 1-shot and 5-shot scenarios on the first two datasets, and is slightly (< 1%) worse on CIFAR-FS. When model ensemble is used, our AC+EnSR almost consistently outperforms existing methods.
More importantly, in the worst-case (ACC 1 ) and near-worst case average metrics (ACC 10 , ACC 100 ), our methods are almost consistently better, and mostly higher by > 2% margins. In the worst-case metrics, model ensemble (AC+EnSR) is again better than single model. And, the ranking from ACC m is significantly different from that based on ACC 1 , but the ranking orders are almost consistent among the 3 worst-case metrics. Hence, the worst-case metric is more stable than the average accuracy metric.
4.3 Relationships: µ, σ, µ − 3σ, ACC 1 and ACC m Table 3 presents a few important evaluation metrics, including the commonly used average accuracy ACC m (µ), the proposed worst-case accuracy ACC 1 that we advocate, the standard deviation σ, and the surrogate µ − 3σ that motivates the proposed method.
As aforementioned, because the worst-case accuracy ACC 1 is unable to be directly modeled or optimized using few-shot samples, we alternatively resort to the surrogate µ − 3σ inspired by the 3σ rule. This objective is again difficult because σ is unable to be directly accessed, and we resort to the bias-variance decomposition: reducing σ and increasing µ simultaneously.
As Table 3 shows, both µ and σ are correlated with ACC 1 , but not tightly correlated. For example, S2M2 R [23] often has the smallest σ, but because its µ (ACC m ) is not the most competitive, its worst-case ACC 1 is significantly lower than ours. We do need to optimize both µ and σ simultaneously.
µ − 3σ, which motivates our method, correlates better with ACC 1 than σ or µ, as Table 3 shows. However, this quantity is still far from being a perfect surrogate. It is still a great challenge to model the worst-case ACC 1 end-to-end.

Ablation Analyses
Switching backbone. Table 4 shows the results of 5-way classification on mini -ImageNet by switching to different backbones. Our adaptability calibration (AC) alone is sometimes harmful in the 1-shot scenario, but always beneficial for 5shot, which coincides well with the observations in Fig. 3.
On the other hand, the proposed stability regularization (SR) is consistently helpful in all cases and all metrics, especially in the 1-shot case. The proposed model ensemble method (EnSR) is almost consistently improving all evaluation metrics on all backbone models, in particular for the difficult 1-shot scenario.
Switching pretraining methods. We further conduct experiments to explore the influence of pretraining methods on the proposed strategies. Specifically, we additionally adopt two pretraining methods from [5], denoted as Baseline and Baseline++, respectively, for simplicity. As shown in Table 5, when pretrained with Baseline, after applying AC and SR, ACC 1 and ACC 10 are increased (on average by 2% in 1-shot and 1% in 5-shot), although ACC m is slightly Table 3. Key evaluation metrics, including the commonly used average accuracy ACCm (µ), the proposed worst-case accuracy ACC1, the standard deviation σ, and the surrogate µ − 3σ that motivates the proposed method. ♢ ACCm from our implementation † ACCm from [5] decreased (about 0.2%). This observation not only shows our AC+SR's effectiveness, but also indicates that average and worst-case accuracy are not aligned in general. With AC+EnSR, the accuracy for all metrics gains significant margins than the naive Baseline method, in particular, 4-5% for 1-shot. When pretraining with Baseline++, the accuracy in all cases are generally improved by gradually deploying AC+SR or AC+EnSR, too. The proposed techniques are compatible with various pretraining methods.
Calibrating adaptability differently. Table 6 shows the results by varying the number of learnable convolutional groups in ResNet 18 to experiment with different calibration of adaptability. First, there is a clear distinction between the first row (W only) and other rows, indicating that it is harmful to freeze the entire backbone and only fine-tune the classification head W .
Second, when more residual groups are made learnable, there is not a clear winner, especially when we consider different evaluation metrics. For example, every row in Table 6's last 3 rows has been the champion for at least one evalua-  tion metric in 5-shot. But, if we consider 1-shot, it is obviously observed that the second row (fine-tuning W +'res5') is significantly better. This is the adaptability calibration we choose in our experiments.
Generalize SR to other data. Since only raw images are required to compute the SR loss, it enjoys the flexibility to use others images (i.e., those not from D b ) for its computation (i.e., being generalizable). We use CUB [34], Cars [18], Describable Textures (DTD) [6], Pets [25], VGG Flower (Flower) [24] and CIFAR-100 [19] as the images to compute our stability regularization loss when the base set is mini -ImageNet. All these images are resized to 84×84.
Results in Table 7 reveals an interesting observation. No matter what images are used to compute the SR loss, the accuracy is consistently higher than the baseline (AC without SR), regardless of which metric is considered. That is, the stability regularization is indeed generalizable.
Moreover, although D b (the base set) performs the best in general, using other images for SR computation leads to results that are on par with it. That is, we do not require images used in this regularization to be visually similar or semantically correlated. Our stability regularization is consistently useful.  Table 7. 5-way 1-shot results on mini-ImageNet by deploying AC+SR with ResNet 18 as the backbone, while the SR loss is computed with images from different datasets, denoted as Dsr. The first row ('−') is the baseline with only AC, and the second row is AC+SR, where SR uses the base set (mini-ImageNet).

Conclusions
This paper advocated to use the worst-case accuracy in optimizing and evaluating few-shot learning methods, which is a better fit to real-world applications than the commonly used average accuracy. Worst-case accuracy, however, is much more difficult to work on. We designed a surrogate loss inspired by the 3σ rule, and in turn proposed two strategies to implicitly optimize this surrogate: a stability regularization (SR) loss together with model ensemble to reduce the variance, and an adaptability calibration (AC) to vary the number of learnable parameters to reduce the bias. The proposed strategies have achieved significantly higher worst-case (and also average) accuracy than existing methods. In the future, we will design more direct attacks to reduce the worst-case error, because the current surrogate is not a highly accurate approximation of the worst-case performance yet.