{'fraction_non_alphanumeric': 0.04147894403693637, 'fraction_numerical': 0.016234128904940984, 'mean_word_length': 4.584944894988563, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Variational Metric Scaling for Metric-Based Meta-Learning

Metric-based meta-learning has attracted a lot of attention due to its effectiveness and efficiency in few-shot learning. Recent studies show that metric scaling plays a crucial role in the performance of metric-based meta-learning algorithms. However, there still lacks a principled method for learning the metric scaling parameter automatically. In this paper, we recast metric-based meta-learning from a Bayesian perspective and develop a variational metric scaling framework for learning a proper metric scaling parameter. Firstly, we propose a stochastic variational method to learn a single global scaling parameter. To better fit the embedding space to a given data distribution, we extend our method to learn a dimensional scaling vector to transform the embedding space. Furthermore, to learn task-specific embeddings, we generate task-dependent dimensional scaling vectors with amortized variational inference. Our method is end-to-end without any pre-training and can be used as a simple plug-and-play module for existing metric-based meta-algorithms. Experiments on mini-ImageNet show that our methods can be used to consistently improve the performance of existing metric-based meta-algorithms including prototypical networks and TADAM.


Introduction
Few-shot learning (Li, Fergus, and Perona 2006) aims to assign unseen samples (query) to the belonging categories with very few labeled samples (support) in each category. A promising paradigm for few-shot learning is meta-learning, which learns general patterns from a large number of tasks for fast adaptation to unseen tasks. Recently, metric-based meta-learning algorithms (Garcia and Bruna 2017; Koch, Zemel, and Salakhutdinov 2015;Snell, Swersky, and Zemel 2017;Vinyals et al. 2016) demonstrate great potential in fewshot classification. Typically, they learn a general mapping, which projects queries and supports into an embedding space. These models are trained in an episodic manner (Vinyals et al. 2016) by minimizing the distances between a query and same-labeled supports in the embedding space. Given a new task in testing phase, a nearest neighbour classifier is applied to assign a query to its nearest class in the embedding space.
Many metric-based meta-algorithms (short form for metalearning algorithms) employ a softmax classifier with crossentropy loss, which is computed with the logits being the distances between a query and supports in the embedding (metric) space. However, it has been shown that the scale of the logits -the metric scaling parameter, is critical to the performance of the learned model. Snell, Swersky, and Zemel (2017) found that Euclidean distance significantly outperforms cosine similarity in few-shot classification, while Oreshkin, López, and Lacoste (2018) and  pointed out that there is no clear difference between them if the logits are scaled properly. They supposed that there exists an optimal metric scaling parameter which is data and architecture related, but they only used cross validation to manually set the parameter, which requires pre-training and cannot find an ideal solution.
In this paper, we aim to design an end-to-end method that can automatically learn an accurate metric scaling parameter. Given a set of training tasks, to learn a data-dependent metric scaling parameter that can generalize well to a new task, Bayesian posterior inference over learnable parameters is a theoretically attractive framework (Gordon et al. 2018;Ravi and Beatson 2019). We propose to recast metric-based meta-algorithms from a Bayesian perspective and take the metric scaling parameter as a global parameter. As exact posterior inference is intractable, we introduce a variational approach to efficiently approximate the posterior distribution with stochastic variational inference.
While a proper metric scaling parameter can improve classification accuracy via adjusting the cross-entropy loss, it simply rescales the embedding space but does not change the relative locations of the embedded samples. To transform the embedding space to better fit the data distribution, we propose a dimensional variational scaling method to learn a scaling parameter for each dimension, i.e., a metric scaling vector. Further, in order to learn task-dependent embeddings (Oreshkin, López, and Lacoste 2018), we propose an amortized variational approach to generate task-dependent metric scaling vectors, accompanied by an auxiliary training strategy to avoid time-consuming pre-training or co-training.
Our metric scaling methods can be used as pluggable modules for metric-based meta-algorithms. For example, it can be incorporated into prototypical networks (PN) (Snell, Swersky, and Zemel 2017) and all PN-based algorithms to improve their performance. To verify this, we conduct extensive experiments on the miniImageNet benchmark for few-shot classification progressively. First, we show that the proposed stochastic variational approach consistently improves on PN, and the improvement is large for PN with cosine similarity. Second, we show that the dimensional variational scaling method further improves upon the one with single scaling parameter, and the task-dependent metric scaling method with amortized variational inference achieves the best performance. We also incorporate the dimensional metric scaling method into TADAM (Oreshkin, López, and Lacoste 2018) in conjunction with other tricks proposed by the authors and observe notable improvement. Remarkably, after incorporating our method, TADAM achieves highly competitive performance compared with state-of-the-art methods.
To sum up, our contributions are as follows: • We propose a generic variational approach to automatically learn a proper metric scaling parameter for metric-based meta-algorithms.
• We extend the proposed approach to learn dimensional and task-dependent metric scaling vectors to find a better embedding space by fitting the dataset at hand.
• As a pluggable module, our method can be efficiently used to improve existing metric-based meta-algorithms.

Related Work
Metric-based meta-learning. Koch, Zemel, and Salakhutdinov (2015) proposed the first metric-based meta-algorithm for few-shot learning, in which a siamese network (Chopra et al. 2005) is trained with the triplet loss to compare the similarity between a query and supports in the embedding space. Matching networks (Vinyals et al. 2016) proposed the episodic training strategy and used the cross-entropy loss where the logits are the distances between a query and supports. Prototypical networks (Snell, Swersky, and Zemel 2017) improved Matching networks by computing the distances between a query and the prototype (mean of supports) of each class. Many metric-based meta-algorithms (Oreshkin, López, and Lacoste 2018;Fort 2017;Sung et al. 2018;Li et al. 2019) extended prototypical networks in different ways. Some recent methods proposed to improve prototypical networks by extracting task-conditioning features. Oreshkin, López, and Lacoste (2018) trained a network to generate task-conditioning parameters for batch normalization. Li et al. (2019) extracted task-relevant features with a category traversal module. Our methods can be incorporated into these methods to improve their performance.
In addition, there are some works related to our proposed dimensional scaling methods. Kang et al. (2019) trained a meta-model to re-weight features obtained from the base feature extractor and applied it for few-shot object detection. Lai et al. (2018) proposed a generator to generate task-adaptive weights to re-weight the embeddings, which can be seen as a special case of our amortized variational scaling method.
Metric scaling. Cross-entropy loss is widely used in many machine learning problems, including metric-based metalearning and metric learning (Babenko and Lempitsky 2015;Ranjan, Castillo, and Chellappa 2017;Liu et al. 2017;Wang et al. 2017;Babenko and Lempitsky 2015;Wan et al. 2018). In metric learning, the influence of metric scaling on the cross-entropy loss was first studied in Wang et al. (2017) and Ranjan, Castillo, and Chellappa (2017). They treated the metric scaling parameter as a trainable parameter updated with model parameters or a fixed hyperparameter.  proposed a "heating-up" scaling strategy, where the metric scaling parameter decays manually during the training process. The scaling of logits in crossentropy loss for model compression was also studied in Hinton, Vinyals, and Dean (2015), where it is called temperature scaling. The temperature scaling parameter has also been used in confidence calibration (Guo et al. 2017).
The effect of metric scaling for few-shot learning was first discussed in Snell, Swersky, and Zemel (2017) and Oreshkin, López, and Lacoste (2018). The former found that Euclidean distance outperforms cosine similarity significantly in prototypical networks, and the latter argued that the superiority of Euclidean distance could be offset by imposing a proper metric scaling parameter on cosine similarity and using cross validation to select the parameter.

Notations and Problem Statement
Let Z = X × Y be a domain where X is the input space and Y is the output space. Assume we observe a meta-sample S = {D i = D tr i ∪ D ts i } n i=1 including n training tasks, where the i-th task consists of a support set of size m, D tr i = {z i,j = (x i,j , y i,j )} m j=1 , and a query set of size q, D ts i = {z i,j = (x i,j , y i,j )} m+q j=m+1 . Each training data point z i,j belongs to the domain Z. Denote by θ the model parameters and α the metric scaling parameter. Given a new task and a support set D tr sampled from the task, the goal is to predict the label y of a query x.

Prototypical Networks
Prototypical networks (PN) (Snell, Swersky, and Zemel 2017) is a popular and highly effective metric-based meta-algorithm. PN learns a mapping φ θ which projects queries and the supports to an M -dimensional embedding space. For each class k ∈ {1, 2, . . . , K}, the mean vector of the supports of class k in the embedding space is computed as the class prototype c k . The embedded query is compared with the prototypes and assigned to the class of the nearest prototype. Given a similarity metric d : R M × R M → R + , the probability of a query z i,j belonging to class k is, . (1) Training proceeds by minimizing the cross-entropy loss, i.e., the negative log-probability − log p θ (y i,j = k|x i,j , D tr i ) of its true class k. After introducing the metric scaling parameter α, the classification loss of the i th task becomes . (2) The metric scaling parameter α has been found to affect the performance of PN significantly.

Stochastic Variational Scaling
In the following, we recast metric-based meta-learning from a Bayesian perspective. The predictive distribution can be parameterized as The conditional distribution p θ (y|x, D tr , α) is the discriminative classifier parameterized by θ. Since the posterior distribution p θ (α|S) is intractable, we propose a variational distribution q ψ (α) parameterized by parameters ψ to approximate p θ (α|S). By minimizing the KL divergence between the approximator q ψ (α) and the real posterior distribution p θ (α|S), we obtain the objective function We want to optimize L(ψ, θ; S) w.r.t. both the model parameters θ and the variational parameters ψ. The gradient and the optimization procedure of the model parameters θ are similar to the original metric-based meta-algorithms (Vinyals et al. 2016;Snell, Swersky, and Zemel 2017) as shown in Algorithm 1.
To derive the gradients of the variational parameters, we leverage the re-parameterization trick proposed by Kingma and Welling (2013) to derive a practical estimator of the variational lower bound and its derivatives w.r.t. the variational parameters. In this paper, we use this trick to estimate the derivatives of L(ψ, θ; S) w.r.t. ψ. For a distribution q ψ (α), we can re-parameterize α ∼ q ψ (α) using a differentiable transformation α = g ψ ( ), if exists, of an auxiliary random variable . For example, given a Gaussian distribution q µ,σ (α) = N (µ, σ 2 ), the re-parameterization is g µ,σ ( ) = σ+µ, where ∼ N (0, 1). Hence, the first term in (4) . We apply a Monte Carlo integration with a single sample α i = g ψ ( i ) for each task to get an unbiased estimator. Note that α i is sampled for the task D i rather than for each instance, i.e., {z i,j } m+q j=1 share the same α i . The second term in (4) can be computed with a given prior distribution p(α). Then, the final objective function is Estimation of gradients. The objective function (5) is a general form. Here, we consider q ψ (α) as a Gaussian distribution q µ,σ (α) = N (µ, σ 2 ). The prior distribution is also a Gaussian distribution p(α) = N (µ 0 , σ 2 0 ). By the fact that the KL divergence of two Gaussian distributions has a closedform solution, we obtain the following objective function where g µ,σ ( i ) = σ i + µ. The derivatives of L(µ, σ, θ; S) w.r.t. µ and σ respectively are In particular, we apply the proposed variational metric scaling method to Prototypical Networks with feature extractor φ θ . The details of the gradients and the iterative update procedure are shown in Algorithm 1. It can be seen that the gradients of the variational parameters are computed using the intermediate quantities in the computational graph of the model parameters θ during back-propagation, hence the computational cost is very low. For meta-testing, we use µ (mean) as the metric scaling parameter for inference.
The proposed variational scaling framework is general. Note that training the scaling parameter α together with the model parameters (Ranjan, Castillo, and Chellappa 2017) is a special case of our framework, when q ψ (α) is defined as N (µ, 0), the variance of the prior distribution is σ 0 → ∞, and the learning rate is fixed as l θ = l ψ .

Dimensional Stochastic Variational Scaling
Metric scaling can be seen as a transformation of the metric (embedding) space. Multiplying the distances with the scaling parameter accounts to re-scaling the embedding space. Figure 1: The middle figure shows a metric space in which the query (blue) and the support samples (red) are normalized to a unit ball. The left and right figures show the spaces scaled by a single parameter α = 1.5 and a two-dimensional vector (α 1 , α 2 ) = (1.5, 0.5), respectively. The query Q is still assigned to class 1 in the left figure but to class 2 in the right one.
By this point of view, we generalize the single scaling parameter to a dimensional scaling vector which transforms the embedding space to fit the data.
If the dimension of the embedding space is too low, the data points cannot be projected to a linearly-separable space. Conversely, if the dimension is too high, there may be many redundant dimensions. The optimal number of dimensions is data-dependent and difficult to be selected as a hyperparameter before training. Here, we address this problem by learning a data-dependent dimensional scaling vector to modify the embedding space, i.e., learning different weights for each dimension to highlight the important dimensions and reduce the influence of the redundant ones. Figure 1 shows a two-dimensional example. It can be seen that the single scaling parameter α simply changes the scale of the embedding space, but the dimensional scaling α = (α 1 , α 2 ) changes the relative locations of the query and the supports.
The proposed dimensional stochastic variational scaling method is similar to Algorithm 1, with the variational parameters µ = (µ 1 , µ 2 , . . . , µ M ) and σ = (σ 1 , σ 2 , . . . , σ M ). Accordingly, the metric scaling operation is changed to The gradients of the variational parameters are still easy to compute and the computational cost can be ignored.

Amortized Variational Scaling
The proposed stochastic variational scaling methods above consider the metric scale as a global scalar or vector parameter, i.e., the entire meta-sample S = {D i } n i=1 shares the same embedding space. However, the tasks randomly sampled from the task distribution may have specific task-relevant feature representations (Kang et al. 2019;Lai et al. 2018;Li et al. 2019). To adapt the learned embeddings to the taskspecific representations, we propose to apply amortized variational inference to learn the task-dependent dimensional scaling parameters.
For amortized variational inference, α is a local latent variable dependent on D instead of a global parameter. Similar to stochastic variational scaling, we apply the variational distribution q ψ(β) (α|D) to approximate the posterior distribution p θ (α|D). In order to learn the dependence between α and D, amortized variational scaling learns a mapping approximated by a neural network G β , from the task D i to the distribution parameters {µ i , σ i } of α i .
By leveraging the re-parameterization trick, we obtain the objective function of amortized variational scaling: where g µi,σi ( i ) = σ i i + µ i . Note that the local parameters {µ i , σ i } are functions of β, i.e., {µ i , σ i } = G β (D i ). We iteratively update β and θ by minimizing the loss function (10) during meta-training During meta-testing, for each task, the generator produces a variational distribution's parameters and we still use the mean vector as the metric scaling vector for inference.

Algorithm 2: Dimensional Amortized Variational Scaling for Prototypical Networks
Input: Meta-sample {D i } n i=1 , learning rates l θ , l β , prior µ 0 , σ 0 and step size l λ . Initialize: β and θ randomly, λ = 1. 1 for i in {1, 2, . . . , n} do Auxiliary loss. To learn the mapping G β from a set D i to the variational parameters of the local random variable α i , we compute the mean vector of the embedded queries and the embedded supports as the task prototype to generate the variational parameters. A problem is that the embeddings are not ready to generate good scaling parameters during early epochs. Existing approaches including co-training (Oreshkin, López, and Lacoste 2018) and pre-training (Li et al. 2019) can alleviate this problem at the expense of computational efficiency. They pre-train or co-train an auxiliary supervised learning classifier in a traditional supervised manner over the meta-sample S, and then apply the pre-trained embeddings to generate the task-specific parameters and fine-tune the embeddings during meta-training. Here, we propose an endto-end algorithm which can improve training efficiency in comparison with pre-training or co-training. We optimize the following loss function (11) where an auxiliary weight λ is used instead of minimizing (10) in Algorithm 2 , i.e., where L(θ; S) = − n i=1 m+q j=m+1 log p θ (y i,j |x i,j , 1), i.e., no scaling is used. Given a decay step size γ, λ starts from 1 and linearly decays to 0 as the number of epochs increases, i.e., λ = λ − 1/γ. During the first epochs, the weight of the gradients ∂L(θ;S) ∂θ is high and the algorithm learns the embeddings of PN. As the training proceeds, β is updated to tune the learned embedding space. See the details in Algorithm 2.

Experiments
To evaluate our methods, we plug them into two popular algorithms, prototypical networks (PN) (Snell, Swersky, andZemel 2017) andTADAM (Oreshkin, López, andLacoste 2018), implemented by both Conv-4 and ResNet-12 backbone networks. To be elaborated later, Table 1 shows our main results in comparison to state-of-the-art meta-algorithms, where it can been that our dimensional stochastic variational scaling algorithm outperforms other methods substantially. For TADAM, we incorporate our methods into TADAM in conjunction with all the techniques proposed in their paper and still observe notable improvement.

Dataset and Experimental Setup
miniImageNet. The miniImageNet (Vinyals et al. 2016) consists of 100 classes with 600 images per class. We follow the data split suggested by Ravi and Larochelle (2017), where the dataset is separated into a training set with 64 classes, a testing set with 20 classes and a validation set with 16 classes.
Model architecture. To evaluate our methods with different backbone networks, we re-implement PN with the Conv-4 architecture proposed by Snell, Swersky, and Zemel (2017) and the ResNet-12 architecture adopted by Oreshkin, López, and Lacoste (2018), respectively.
The Conv-4 backbone contains four convolutional blocks, where each block is sequentially composed of a 3 × 3 kernel convolution with 64 filters, a batch normalization layer, a ReLU nonlinear layer and a 2 × 2 max-pooling layer.
The ResNet-12 architecture contains 4 Res blocks, where each block consists of 3 convolutional blocks followed by a 2 × 2 max-pooling layer.
Training details. We follow the episodic training strategy proposed by (Vinyals et al. 2016). In each episode, K classes and N shots per class are selected from the training set, the validation set or the testing set. For fair comparisons, the number of queries, the sampling strategy of queries, and the testing strategy are designed in line with PN or TADAM.
For Conv-4, we use Adam optimizer with a learning rate of 1e − 3 without weight decay. The total number of training episodes is 20, 000 for Conv-4. And for ResNet-12, we use SGD optimizer with momentum 0.9, weight decay 4e−4 and 45, 000 episodes in total. The learning rate is initialized as 0.1 and decayed 90% at episode steps 15000, 30000 and 35000. Besides, we use gradient clipping when training ResNet-12. The reported results are the mean accuracies with 95% confidence intervals estimated by 5 runs.
We normalize the embeddings before computing the distances between them. As shown in Eq. (7) and (8), the gradient magnitude of variational metric scaling parameters is proportional to the norm of embeddings. Therefore, to foster the learning process of these parameters, we adopt a separate learning rate l ψ for all variational metric scaling parameters.

Evaluation
The effectiveness of our proposed methods is illustrated in Table 2 progressively, including stochastic variational scaling (SVS), dimensional stochastic variational scaling (D-SVS) miniImageNet test accuracy Backbones Model 5-way 1-shot 5-way 5-shot

ResNet
Performance of SVS. We study the performance of SVS by incorporating it into PN. We consider both 5-way and 20way training scenarios. The prior distribution of the metric scaling parameter is set as p(α) = N (1, 1) and the variational parameters are initialized as µ init = 100, σ init = 0.2. The learning rate is set to be l ψ = 1e − 4.
Results in Table 3 show the effect of the metric scaling parameter (SVS). Particularly, significant improvement is observed for the case of PN with cosine similarity and for the case of 5-way 1-shot classification. Moreover, it can be seen that with metric scaling there is no clear difference between the performance of Euclidean distance and cosine similarity.
We also compare the performance of a fixed σ = 0.2 with a trainable σ. We add a shifted ReLU activation function (x = max{1e−2, x}) on the learned σ to ensure it is positive. Nevertheless, in our experiments, we observe that the training is very stable and the variance is always positive even without the ReLU activation function. We also find that there is no significant difference between the two settings. Hence, we treat σ as a fixed hyperparameter in other experiments.
Performance of D-SVS. We validate the effectiveness of D-SVS by incorporating it into PN and TADAM, with the results shown in Table 1 and Table 2. On 5-way-1-shot classification, for PN, we observe about 4.90% and 1.41% absolute increase in test accuracy with Conv-4 and ResNet-12 respectively; for TADAM, 1.66% absolute increase in test accuracy is observed. The learning rate for D-SVS is set to be l ψ = 16.
Here we use a large learning rate since the gradient magnitude of each dimension of the metric scaling vector is extremely small after normalizing the embeddings.
Performance of D-AVS. We evaluate the effectiveness of D-AVS by incorporating it into PN. We use a multi-layer perception (MLP) with one hidden layer as the generator G β . The learning rate l β is set to be 1e − 3. In Table 2, on both 5-way 1-shot and 5-way 5-shot classification, we observe about 1.0% absolute increase in test accuracy for dimensional amortized variational scaling (D-AVS) over SVS with a single scaling parameter. In our experiments, the hyperparameter γ is selected from the range of [100, 150] with 200 training epochs in total.
Ablation study of D-AVS. To assess the effects of the auxiliary training strategy and the prior information, we provide an ablation study as shown in Table 4. Without the auxiliary training and the prior information, D-AVS degenerates to a task-relevant weight generating approach (Lai et al. 2018).   Table 4: Ablation study of prototypical networks with D-AVS by our re-implementation using Conv-4.
Noticeable performance drops can be observed after removing the two components. Removing either one of them also leads to performance drop, but not as significant as removing both. The empirical results confirm the necessity of the auxiliary training and a proper prior distribution for amortized variational metric scaling.

Robustness Study
We also design experiments to show: 1) The convergence speed of existing methods does not slow down after incorporating our methods; 2) Given the same prior distribution, the variational parameters converge to the same values in spite of different learning rates and initializations.
For the iterative update of the model parameters θ and the variational parameters ψ, a natural question is whether it will slow down the convergence speed of the algorithm. Figure 2 shows the learning curves of PN and PN+D-SVS on both 5-way 1-shot and 5-way 5-shot classification. It can be seen that the incorporation of SVS does not reduce the convergence speed.
We plot the learning curves of the variational parameter µ w.r.t. different initializations and different learning rates l ψ . Given the same prior distribution µ 0 = 1, Fig. 3(a) shows that the variational parameter µ with different initializations will converge to the same value. Fig. 3(b) shows that µ is robust to different learning rates.

Conclusion
In this paper, we have proposed a generic variational metric scaling framework for metric-based meta-algorithms, under which three efficient end-to-end methods are developed. To learn a better embedding space to fit data distribution, we have considered the influence of metric scaling on the embedding space by taking into account data-dependent and task-dependent information progressively. Our methods are lightweight and can be easily plugged into existing metricbased meta-algorithms to improve their performance.