{'fraction_non_alphanumeric': 0.03212989306386736, 'fraction_numerical': 0.019228882566467185, 'mean_word_length': 4.829854160709179, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multi-level Distance Regularization for Deep Metric Learning

We propose a novel distance-based regularization method for deep metric learning called Multi-level Distance Regularization (MDR). MDR explicitly disturbs a learning procedure by regularizing pairwise distances between embedding vectors into multiple levels that represents a degree of similarity between a pair. In the training stage, the model is trained with both MDR and an existing loss function of deep metric learning, simultaneously; the two losses interfere with the objective of each other, and it makes the learning process difficult. Moreover, MDR prevents some examples from being ignored or overly influenced in the learning process. These allow the parameters of the embedding network to be settle on a local optima with better generalization. Without bells and whistles, MDR with simple Triplet loss achieves the-state-of-the-art performance in various benchmark datasets: CUB-200-2011, Cars-196, Stanford Online Products, and In-Shop Clothes Retrieval. We extensively perform ablation studies on its behaviors to show the effectiveness of MDR. By easily adopting our MDR, the previous approaches can be improved in performance and generalization ability.


Introduction
Deep Metric Learning (DML) aims to learn an appropriate metric that measures the semantic difference between a pair of images as a distance between embedding vectors. Many research areas such as image retrieval (Sohn 2016;Yuan, Yang, and Zhang 2017;Oh Song et al. 2017;Duan et al. 2018;Ge 2018) and face recognition (Wang et al. 2017;Liu et al. 2017;Wang et al. 2018;Deng et al. 2019) are based on DML to seek appropriate metrics among instances. Those studies focus on devising a better loss function for DML.
Most of previous loss functions (Sohn 2016;Bromley et al. 1994;Hadsell, Chopra, and LeCun 2006;Yi et al. 2014;Hoffer and Ailon 2015;Schroff, Kalenichenko, and Philbin 2015) use binary supervision that indicates whether a given pair is positive or negative. Their common objective is to minimize the distance between a positive pair and maximize the distance between a negative pair ( Figure 1a). However, without any constraints, a model trained with such objective is prone to overfitting on a training set because positive pairs can be aligned too closely while the negative pairs can be aligned too far in the embedding space. Therefore, several loss functions employ additional terms to avoid positive pairs to be too close and negative pairs to be too far, e.g., margin m in Triplet loss (Schroff, Kalenichenko, and Philbin 2015) and Constrastive loss (Hadsell, Chopra, and LeCun 2006). Despite these attempts, they still can suffer from overfitting due to the lack of explicit regularization for the distances.
Our insight is that a learning procedure of DML can be enhanced by explicitly regularizing the distance between pairs to disturb a loss function of DML from optimizing an embedding network; one easy way to constrain a distance is to pull the value of the distance to a predefined level. Conventional loss functions of DML adjust the distance according to its label, on the other hand, explicit distance-based regularization prevents the distance from deviating from the predefined level. Those two interfere with the objective of each other, thus it makes the learning process difficult and allows the embedding network to be more robust for generalization. Additionally, we consider multiple levels with disjoint intervals to regularize distances, not a single level, because a degree of inter-class similarity or intra-class variation can be different depending on classes or instances.
We propose a novel method called Multi-level Distance Regularization (MDR) that makes the conventional loss functions of DML have difficulty in converging by holding each distance so that it does not deviate from the belonging level. At first, MDR normalizes pairwise distances among the embedding vectors of a mini-batch, with their mean and standard deviation to obtain the objective degree of similarity between a pair by considering overall distribution. MDR defines the multiple levels that represent various degrees of similarity for pairwise distances, and the levels and the belonging distances are trained to approach each other ( Figure  1b). A conventional loss function of DML struggles to optimize a model by overcoming the disturbance from the proposed regularization. Therefore, the learning process succeeds in learning a model with a better generalization ability. We summarize our contributions: • We introduce MDR, a novel regularization method for DML. The method disturbs optimizing pairwise distances by preventing them from deviating from its belonging level for better generalization.  (Schroff, Kalenichenko, and Philbin 2015), which is one of the representative conventional learning. It increases the relative difference between distances of a positive pair and that of a negative pair more than margin m. (b) illustrates our learning combined with the triplet learning. It has multiple levels with disjoint intervals to reflect various degrees of similarity between pairs. It disturbs the learning procedure to construct an efficient embedding space by preventing the pairwise distances from deviating from its belonging level.
• MDR achieves the-state-of-the-art performance on various benchmark datasets (Wah et al. 2011;Krause et al. 2013;Oh Song et al. 2016;Liu et al. 2016) of DML. Moreover, our extensive ablation studies show that MDR can be adopted to any backbone networks and any distance-based loss functions to improve the performance of a model.

Related Work
Loss Function. Improving the loss function is one of the key objectives in recent DML studies. One family of loss functions (Sohn 2016;Bromley et al. 1994;Schroff, Kalenichenko, and Philbin 2015;Oh Song et al. 2016;Wang et al. 2019a;Wu et al. 2017) focuses on optimizing pairwise distance between instances. The common objective of these functions is to minimize the distance between positive pairs and to maximize the distance between negative pairs in an embedding space. Contrastive loss (Bromley et al. 1994) samples pairs of two instances, whereas Triplet loss (Schroff, Kalenichenko, and Philbin 2015) samples triplets of anchor, positive and negative instances; then both losses optimize the distance between the sampled instances. Also, Global Loss (Kumar BG, Carneiro, and Reid 2016)  Figure 2: Learning procedure of the proposed MDR. The embedding network generates embedding vectors from given images. Our MDR computes a matrix of pairwise distances for the embedding vectors, and then, the distances are normalized after vectorization. In our learning scheme, a model is trained by simultaneously optimizing the conventional metric learning loss such as Triplet loss (Schroff, Kalenichenko, and Philbin 2015) and the proposed loss, which regularizes the normalized pairwise distances with multiple levels.

Proposed Method
In this section, we introduce a new regularization method called Multi-level Distance Regularization (MDR), which makes the learning procedure difficult by preventing each pairwise distance from deviating from a corresponding level, to learn a robust feature representation.

Multi-level Distance Regularization
We describe the detailed procedure of MDR to regulate pairwise distances in three steps ( Figure 2).
(1) Distance Normalization. This step is performed to obtain an objective degree of distance by considering overall distribution for stable regularization. Here, an embedding network f maps an image x into an embedding vector e with a certain dimensionality: e = f (x). A distance is defined as Euclidean distance between two given embedding vectors, d(e i , e j ) = e i − e j 2 . We normalize the distance as: where µ is mean of distances and σ is standard deviation of distances a set of pairs, which is P = {(e i , e j )|i = j} for all instances of a mini-batch. To more widely consider the overall dataset, we employ the momentum updates: where µ * t and σ * t are respectively the momented mean and momented standard deviation at iteration t, and γ is the momentum. With the momented statistics, the normalized distance is re-written: (2) Level Assignment. MDR designates a level that acts as a regularization goal for each normalized distance. We define a set of levels s ∈ S, and the levels are initialized with predefined values; each level s is interpreted as a multiplier of the standard deviation of the normalized distance. g(d; s) is an assignment function that outputs whether the given distance d and the given level s are the closest or not, and is defined as: By adopting the assignment function, MDR selects valid regularization levels for each distance with the consideration of various degrees of similarities.
(3) Regularization. Finally, this step is performed to prevent pairwise distances from deviating from its belonging level. MDR minimizes the difference between a given normalized pairwise distance and the assigned level: (5) The levels are learnable parameters and are updated to optimally regularize the pairwise distances. Each normalized distance is trained to become closer to the assigned level; the assigned level is also trained to become closer to the corresponding distances. As iterations pass, the levels are trained to properly divide the normalized distances into multiple intervals. Each level is a representative value of a certain interval in the normalized distance. We describe the initial configuration of the levels in Section 4.3.
In conclusion, MDR has two functional effects of regularization: (1) the multiple levels of MDR disturbs optimizing the pairwise distances among examples, (2) the outermost levels of MDR prevents the positive pairs from getting too close and the negative pairs from getting too far. By the formal effect, the learning process does not easily suffer from overfitting. By the latter effect, the learning process does not suffer from diminishing of the loss from easy examples, and also, does not suffer from being too biased to certain examples such as hard examples. Therefore, MDR stabilizes the learning procedure to achieve a better generalization ability on a test dataset.

Learning
Loss Function. The proposed MDR can be applied to any loss functions L DML such as Contrastive loss    (Schroff, Kalenichenko, and Philbin 2015) and Margin loss (Wu et al. 2017). We mostly adopted Triplet loss as baseline for our experiments: where T is a set of triplets of an anchor e a , a positive e p , and a negative e n sampled from a mini-batch. m is a margin. The final loss function L is defined as the sum of L DML and L MDR with a multiplier λ that balances the losses: L DML optimizes the model by minimizing the distance of positive pairs and maximizing the distance of negative pairs. L MDR regularize the pairwise distances by constraining the distances with multiple levels. The embedding network is trained simultaneously with different objectives. Embedding Normalization Trick for MDR. In our learning procedure, L 2 Normalization (L 2 Norm) is not adopted because it can disturb the proper regularization effect of MDR. However, the lack of L 2 Norm can cause difficulty in finding appropriate hyper-parameters of L DML such as margin m in Triplet loss, because any prior knowledge of the scale of embedding vectors is not given. To overcome the difficulty, we normalize L DML by dividing the embedding vectors e by µ during the training stage, such that the expected pairwise distance is one: E d( ei µ , ej µ ) = 1. We adopt this trick on several loss functions such as Constrastive loss (Hadsell, Chopra, and LeCun 2006), Margin loss (Wu et al. 2017), and Triplet loss in our experiments.

Experiments
To show the effectiveness of MDR and its behaviors, we extensively perform ablation studies and experiments. We follow the standard evaluation protocol and data splits proposed in (Oh Song et al. 2016). For an unbiased evaluation, we conduct 5 independent runs for each experiment and report the mean and the standard deviation of them.

Implementation Details
Embedding Network. All the compared methods and our method use the Inception architecture with Batch Normalization (IBN) (Ioffe and Szegedy 2015) as a backbone network. IBN is pre-trained for ImageNet ILSVRC 2012 dataset (Deng et al. 2009) and then fine-tuned on the target dataset. We attach a fully-connected layer, where its output activation is used as an embedding vector, after the last pooling layer of IBN. For models trained with MDR, L 2 Norm is not applied to the embedding vectors because it disturbs the effect of the regularization. For a fair comparison with the conventional implementation of Triplet loss (Schroff, Kalenichenko, and Philbin 2015) that is used as a baseline, we apply L 2 Norm to those models. Learning. We employ Adam (Kingma and Ba 2014) optimizer with a weight decay of 10 −5 . For CUB-200 and Cars-196, a learning rate and the size of mini-batch are set to 5 · 10 −5 and 128. For SOP and In-Shop, a learning rate and the size of mini-batch are set to 10 −4 and 256. We mainly apply our method to Triplet loss (Schroff, Kalenichenko, and Philbin 2015). As a triplet sampling method, we employ the distance weighted sampling (Wu et al. 2017). The margin m of Triplet loss is set to 0.2. We summarized the hyperparameters of MDR: the configuration of the levels is initialized to three levels of {−3, 0, 3}, and the momentum γ is set to 0.9. λ is set differently for each dataset: 0.6 for CUB-200, 0.2 for Cars-196 and 0.1 for SOP and In-Shop. For most of the datasets, λ of 0.1 is enough to improve a given model; on CUB-200, a strong regularization is more effective because it is a small dataset with only 5,864 training images where a model may easily suffer from overfitting. Those hyper-parameters are not very sensitive to tune, and we explain the effects of each hyper-parameter in the ablation studies at Section 4.3. Image Setting. During training, we follow the standard image augmentation process (Oh Song et al. 2016;Wang et al. 2019a) with the following order: resizing to 256 × 256, random cropping, random horizontal flipping, and resizing to 224 × 224. For evaluation, images are center-cropped.

Comparison with State-of-the-art Methods
We show the comparison of MDR and the recent stateof-the-art methods (Table 1). All compared methods use embedding vectors of 512 dimensionality. Our baseline model is trained by Triplet loss without L 2 Norm (Triplet) and we also report the conventional Triplet with L 2 Norm (Triplet+L 2 Norm). The lack of constraints of L 2 Norm on the embedding space results in poor generalization performance, and it is known that Triplet loss is effective when L 2

CUB-200
Cars Norm is applied (Schroff, Kalenichenko, and Philbin 2015). However, the models with MDR outperform the Triplet+L 2 Norm models on all the datasets. Those results prove the effectiveness of the proposed distance-based regularization.
Experimental Results. MDR improves performance on all the datasets, and, in particular, the improvements are significantly high on the small-sized datasets. For CUB-200, MDR improves 3.7 percentage points on Recall@1 compared to the conventional Triplet+L 2 Norm; the result is 11.5 percentage points higher than Recall@1 of the Triplet. For Cars-196, MDR improves 8.7 percentage points on Re-call@1 compared to the conventional Triplet+L 2 Norm; the result is 12.3 percentage points higher than Recall@1 of the Triplet. MDR also improves the recall performance compared to the baselines on SOP and In-Shop. Moreover, our method significantly outperforms the other state-of-the-art methods in all recall criteria for all datasets.

Ablation Studies
We extensively perform ablation studies on the behaviors of the proposed MDR. Backbone Network. MDR can be widely applicable to any backbone networks (Table 2a). We apply MDR on IBN (Ioffe and Szegedy 2015), ResNet18 (R18) and ResNet50 (He et al. 2016) (R50), and achieve significant improvements for all backbone networks. Especially, a light-weight backbone, R18, with MDR even outperforms the baseline models with a heavy-weight backbone such as R50 and IBN on both datasets. Loss Function. Our MDR also can be widely applicable to any distance-based loss function (Table 2b). We apply MDR on Constrastive loss (Hadsell, Chopra, and LeCun 2006), Margin loss (Wu et al. 2017) and Triplet loss. MDR achieves significant improvements for all loss functions. Level Configuration S. Even though the levels are learnable, we should properly set the number of levels and the initial values of levels. We perform experiments on various initial configurations of levels and validate the importance of the learnability of levels (Table 2c). From the experiments, we find that a sufficiently spaced configuration is better than a tightly spaced configuration; {−3, 0, 3} is

Discussion
Effectiveness in Small Dimensionality. We perform an experiment on various dimensionalities of embedding vector such as 64, 128, 256, and 512. MDR significantly improves the Recall@1 of the models, especially in small dimensionality. In the experiment, our MDR only with 64 dimensionality is similar to or surpasses the performance of other methods with 512 dimensionality (Figure 3a). The result indicates that our MDR constructs a highly efficient embedding space in compact dimensionality. Moreover, the improvements are larger compared to Triplet+L 2 Norm for all dimensionality. Prevention of Overfitting as Regularizer. We investigate the learning curves of three models: Triplet, Triplet+L 2 Norm and Triplet+MDR (Figure 3b). There are two crucial observations: (1) on the training set, Triplet+MDR is less overfitted than the other two methods, but it shows the most high performance on the test set., (2) the recall of Triplet+MDR does not drop until the end of learning, un- like the other methods, which suffer from severe overfitting. These observations indicate that our MDR is an effective regularizer for DML. Equalizing the Two-Norm of Embedding Vectors. We find that the embedding vectors of a model trained with MDR have almost the same two-norm (Figure 4a and 4b).
This shows that the embedding vectors are almost located on a hypersphere, even though the model is trained without L 2 Norm. Therefore, the model trained with MDR achieves similar performance even if L 2 Norm is applied at inference time (Table 3). This observation implies that MDR has similar effects of L 2 Norm at the end of the training, even though MDR is a distance-based regularization and L 2 Norm is norm-based regularization. Discriminative Representation. To show the effectiveness of our method, we visualize how MDR constructs an embedding space. In the embedding space of Triplet and Triplet+L 2 Norm, the class centers are often aligned closely to each other (Figure 5a and 5b). However, in an embedding space of Triplet+MDR, the class centers are evenly spaced with a large margin (Figure 5c). This result indicates that MDR constructs a more discriminative representation than the conventional methods.
Qualitative Analysis on Level Assignment. In the step of the level assignment, a lower level indicates that the pairs are closely aligned in the embedding space and vice versa. Most of the positive pairs are belonging to between level 1 and 2, and most of the negative pairs are belonging to between level 2 and 3. However, hard-positive pairs may belong to level 3 while hard-negative also may belong to level 1 (Figure 6). Therefore, levels are assigned to each pair regardless of given binary supervision. The learning procedure tried to overcome the disturbance that pulls the distances to belonging levels by considering the various degrees of distances; this multi-level disturbance leads to the improvement of the generalization ability.

Conclusion
We introduce a new distance-based regularization method that elaborately adjusts the pairwise distance into multiple levels for better generalization. We prove the effectiveness of MDR by showing the improvements that greatly exceed the existing methods, and by extensively performing the ablation studies of its behaviors. By applying our MDR, many methods can be significantly improved without any extra burdens at inference time.