{'fraction_non_alphanumeric': 0.03384891279982067, 'fraction_numerical': 0.01877381752970186, 'mean_word_length': 5.038917089678511, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Exploring Instance Relations for Unsupervised Feature Embedding

Despite the great progress achieved in unsupervised feature embedding, existing contrastive learning methods typically pursue view-invariant representations through attracting positive sample pairs and repelling negative sample pairs in the embedding space, while neglecting to systematically explore instance relations. In this paper, we explore instance relations including intra-instance multi-view relation and inter-instance interpolation relation for unsupervised feature embedding. Specifically, we embed intra-instance multi-view relation by aligning the distribution of the distance between an instance's different augmented samples and negative samples. We explore inter-instance interpolation relation by transferring the ratio of information for image sample interpolation from pixel space to feature embedding space. The proposed approach, referred to as EIR, is simple-yet-effective and can be easily inserted into existing view-invariant contrastive learning based methods. Experiments conducted on public benchmarks for image classification and retrieval report state-of-the-art or comparable performance.


Introduction
Unsupervised feature embedding methods have attracted increased attention in recent years. As a mainstream method for unsupervised feature embedding, contrastive learning method focuses on learning an embedding function by attracting positive pairs and repelling negative pairs. However, it is challenging to discover positive and negative samples under unsupervised scenarios.
To discover reliable positive and negative samples on unlabeled datasets, instance discrimination based unsupervised methods have achieved promising results on unsupervised feature embedding. Instance Recognition (IR) [Wu et al., 2018] proves that non-parametric instance-wise classification can capture apparent visual similarity. Invariant and Spreading Instance Feature (ISIF) ] exploits data augmentation invariant and instance spreading property for unsupervised learning. Momentum Contrast (MoCo) [He et al., 2020] build a dynamic dictionary with a queue and a moving-averaged encoder. After that, SimCLR [Chen et al., 2020] simplifies these instance discrimination based contrastive learning algorithms without requiring specialized architectures. Compared with instance discrimination based unsupervised methods, supervised learning methods absorb more information from the different instances with same semantic categories. While instance discrimination based methods have shown their effectiveness on several benchmarks, they severely rely on single instance discrimination and data augmentation, and ignore the relations of instances, Figure 1. Augmentation Invariant and Spreading Instance Feature (aISIF)  improves ISIF by two featurelevel augmentation strategies including negative augmentation with interpolation and positive augmentation with extrapolation. However, the instance relations are still not systematically explored, which makes the learned feature embedding less discriminative. Figure 2: Overview of EIR, which includes intra-instance multi-view relation (left) and inter-instance interpolation relation (right). Different shapes represent different semantic categories, and different colors represent different instances.

Inter-instance
In this paper, we present exploring instance relations (EIR) including both intra-instance multi-view relation and interinstance interpolation relation for unsupervised feature embedding, Figure 2. Specifically, for exploring more sophisticated intra-instance relations, we align two distributions with KL divergence, each of which is the distance distribution of an augmented sample with respect to all samples in the training set, Figure 1(a). Benefiting from current data augmentation strategies such as mixup  and cutmix [Yun et al., 2019], we explore inter-instance interpolation relation by transferring the ratio of information for image sample interpolation from pixel space to feature space, Figure 1(b). By simultaneously incorporating these two relations, our approach improves the capability of discrimination in the embedding space. Our approach is simple-yet-effective and can be easily inserted into existing view-invariant contrastive learning based methods.
We present extensive evaluations of our approach on various datasets and tasks to demonstrate its effectiveness. In particular, our method achieves the accuracy of 89.2% on CIFAR-10 in the kNN setting, which is the state-of-the-art performance to the best of our knowledge.
The contributions of this work are summarized as follows: 1. We propose EIR to systematically explore previous overlooked instance relations for unsupervised feature embedding, which is a novel attempt to break through the limit of instance discrimination.
2. We explore the intra-instance multi-view relation by aligning two augmented samples' distance distribution, which measures the relations of the augmented samples with respect to all samples in the training set.
3. We explore inter-instance interpolation relation by transferring the ratio of information for image sample interpolation from pixel space to feature space.

Related Works
There are two related concepts in unsupervised deep learning methods: unsupervised representation learning and unsupervised embedding learning. Unsupervised Representation Learning. Unsupervised representation learning aims at learning 'immediate' conv features, then the features can be used as the initialization for downstream tasks such as image classification and object detection. Some works focus on designing pretext tasks by hiding some information such as context prediction [Doersch et al., 2015] [Tian et al., 2020] proposes to maximize the mutual information between different views and scales to any number of views. MoCo [He et al., 2020] builds a dynamic dictionary with a queue and a moving-averaged encoder. After that, Sim-CLR [Chen et al., 2020] simplifies these instance discrimination based contrastive learning algorithms without requiring specialized architectures or memory bank. For group-based methods, [Yang et al., 2016]  Unsupervised Embedding Learning. Unsupervised embedding learning focuses on learning a low-dimension feature embedding by minimizing the distance of positive pairs and maximizing the difference of negative pairs, and the learned conv features also can be used for initialization. Mining on Manifolds [Iscen et al., 2018] proposes an unsupervised method to mine hard positive and negative samples based on manifold space. ISIF  proposes to utilize the instance-wise supervision to approximate the positive concentration and negative separation properties, and aISIF  improves ISIF by two feature-level augmentation strategies including negative augmentation with interpolation and positive augmentation with extrapolation. PSLR  incorporates adaptive softmax embedding in graph latent space. Current contrastive based unsupervised representation learning methods, i.e. IR [Wu et al., 2018], MoCo [He et al., 2020], can also be applied for unsupervised embedding learning.

Preliminaries
Given an imagery data set X = {x 1 , x 2 , x 3 , · · · , x N } without any manual annotations, the learned embedding model with parameters θ maps an input image x i to a D-dimension embedding in feature space V(t) at t-th epoch, as v i (t) = f θt (x i ). The objective is to learn an embedding feature space where embedding for positive pairs are attracted while negative pairs are repelled. Details will be illustrated as follows.
IR [Wu et al., 2018] proves that CNNs can capture apparent similarity and learn class discriminative feature representation with solely instance-level supervision. The probability of input x being recognized as i-th example is where v = f θ (x), N is the size of whole dataset and τ is a temperature parameter. Objective function is defined as . IR maintains a memory bank for storing the global features of dataset, v i and v j in Eq.(1) denote the i-th and j-th features in memory bank. Memory bank is initialized as unit random vectors and evolves during train- Based on the analysis of IR, we modify instance recognition by applying multi-view data augmentation, which termed IRaug. The probability of an input x and its another viewx being recognized as i-th example is . (2) IRaug forces v andv to be close to the corresponding feature v i in memory bank, is not just implementing IR twice. More specifically, it forces v andv to be close to each other implicitly. We optimize the objective function by We update memory bank with either of v orv. IRaug will serve as our baseline in the following experiments.

Intra-instance Multi-view Relation
From the perspective of ISIF  and MoCo [He et al., 2020], view-invariant contrastive loss focuses on attracting different views of the same instance while repelling the different instances. However, more sophisticated intrainstance relations are not fully explored. One such relation is that the current instance's two augmented samples should have similar distances with each sample in the training set. As illustrated in the right part of Figure 2. Concretely, other than only pushing negative samples far away, two augmented samples from the same instance should push the same negative sample to similar distances. Formally, for the two augmented samples from the same instance, two corresponding distance distributions can be obtained: D andD, andD = {d i =v T v i |i = 1, 2, 3, ..., N }.
(5) These two distributions should be aligned. We define this relation as intra-instance multi-view relation, which can be modeled by the Kullback-Leibler divergence [Kullback, 1959] of distributions D andD. By this way, the objective function is formed by where D(p) denotes the distant distribution of current instance, andD(p) is that of another view.

Inter-instance Interpolation Relation
To address inter-instance relation, aISIF  explores feature-level interpolation as negative samples by v p− = 2 (αv i + (1 − α)v k ), which consistently improves the performance. Inspired by this work, we move one more step to explore the inter-instance relation from pixel level to feature level. Our hypothesis is that better features can be learned if the consistency of the pixel-level interpolation and the featurelevel addition is maintained. As illustrated in the left part of Figure 2, if an image is composed of two images with   Figure 3 shows some possible interpolation alternatives and its performance. For two random instances x i and x j in unlabelled dataset, we can extract partial information from x i with ratio r and partial information from x j with ratio (1 − r) to form a new image sample. The feature embedding of interpolated sample is defined as where ⊕ denotes interpolation operation, and r denotes the ratio of information for interpolation. Meanwhile, the feature addition in embedding space is generated bỹ where 2 denotes 2 normalization. By attracting these two features, the information ratio can be transferred from pixel space to feature space. The inter-instance interpolation relation loss is obtained by In summary, the overall loss function is a combination of the baseline loss L IRaug , the intra-instance multi-view relation loss L intra and the inter-instance interpolation relation loss L inter , which is formulated as Loss = L IRaug + λ 1 L intra + λ 2 L inter .  [Wah et al., 2011] into two parts. The first 100 classes with 5,864 images are used for training, and the last 100 classes with 5,924 images are used for testing, and all images are resized to 224 × 224. We then split the 196 classes of Car-196 [Krause et al., 2013] into two parts. The first 98 classes with 8,054 images are used for training, other 98 classes with 8,131 images are uased for testing, and all images are resized to 224 × 224.
Hyper-parameters. We adopt ResNet18 [He et al., 2016] as the backbone, and initialize the learning rate as 0.03 with the decreasing strategy that the rate is scaled down by 0.1 at 120-th epoch and 0.01 at 160-th epoch. We set momentum to 0.9, weight decay to 0.0005, batch size to 128, temperature parameter τ to 0.1, and the embedding feature space dimension to 128. All experiments are implemented with 2 GeForce RTX 2080Ti GPUs.
Evaluation Metrics. We use a single center-croped image for feature extraction in testing phase, and use linear classification and kNN to evaluate features on classification task. Table 3: Evaluation on STL-10 dataset with ResNet-18 by performing linear classifier (LC) and kNN (k=200) top-1 accuracy. Performance of other methods is copied from PSLR  and aISIF .

Methods
Pre The retrieval performance is evaluated by the probability of correct matching in the top-k retrieval ranking list.

Ablation Study
We take IRaug as our baseline method, then analyze the effect of our proposed module. In Table 1, the baseline method achieves 83.4% top-1 kNN (k=200) accuracy on CIFAR-10 dataset with ResNet18. The experimental results show that compared with the baseline, intra-instance multi-view relation improves the accuracy by 2.9%, inter-instance interpolation relation improves the accuracy by 2.1%, moreover, combining these two components can improve the accuracy by 4.2%.

Image Classification
After fully training networks, we then evaluate the quality of the learned visual representations and feature embedding on image classification. The backbone is ResNet18 and the classification is evaluated by kNN and linear classification. The kNN classification uses feature embedding directly. Linear classification (LC) needs to train a fully connected linear classifier which is initialized from scratch, and the convolution layers are frozen. Considering the difference of image  resolution for different datasets, we evaluate the performance on datasets from low resolution to high resolution. As shown in Table 2, on CIFAR-10 dataset under 1-round training with k equal to 5/20/200, our proposed method consistently achieves the best performance. After 5-rounds training, our method achieves 89.2% for k = 20, which surpasses all other published methods. Moreover, the performance of our method with only 2-rounds training outperforms those of other methods with 5-rounds training. Because the performance of SimCLR [Chen et al., 2020] relies on large batch size, MLP head and stronger data augmentation, it is infeasible to compare EIR with SimCLR under the similar experiment settings. The comparison of performance for different backbone is shown in the supplementary material.
We then conduct experiments on STL-10 dataset, and report LC results with conv5 layer and kNN top-1 results with k = 200 on 5K labeled set under 5K and 105K pre-training respectively. As shown in Table 3, our method achieves 76,4% accuracy with LC and 78.2% accuracy with kNN, which outperforms state-of-the-art aISIF  by 4.2% on 5K pre-train setting. For 105K pre-train setting, our method achieves 84.7% accuracy with kNN, which outperforms aISIF by 0.8%. With LC, aISIF is superior to our method probably because our method needs more training iterations for convergence.
For ImageNet-100 evaluation, we report the LC results on conv5 layer and kNN top-1 results with k = 200. As shown in Table 4, our method achieves 66.4% and 66.5% accuracy for LC and KNN respectively, which outperforms MoCo at least 2% accuracy and surpasses other methods with large margins.
For more challenging fine-grained unseen dataset, our method achieves 14.6%, 22.1%, 32.6% and 44.1% accuracy on CUB-200 dataset and 28.5%, 38.1%, 49.5% and 62.1% accuracy on Car-196 dataset, which achieves stat-of-the-art performance. The retrieval performance on Car-196 surpasses the performance in CUB-200 with large margin, which may result from that birds are more hard to discriminate than cars.

Visualization
To show the effectiveness of positive pairs attracting while negative pairs repelling, we visualize the embedding features on test set of CIFAR-10 by embedding the feature embedding into 2-dimensional space with t-SNE [ Van der Maaten and Hinton, 2008]. Figure 5 shows that instance discrimination based methods achieve the goal that positive attracting while negative pairs repelling, and our method is more discriminative. To help qualitatively illustrate the effectiveness of our proposed method, Figure 6 shows the 8 nearest neighbourhood retrieval results on Car-196 test set including success and failure cases. The first column is 4 query images, and each one follows 2 rows retrieval results including IRaug (top) and EIR (down), and each row is ordered by cosine similarity. The 8 nearest neighbours of the 4-th query image are all false positive, they share high similarity on appearance, color and posture. However, from the results of first 3 query images, we can find that our model can discover positive samples with different colors, which shows that the features learned by our method focus on high-level semantic information.

Conclusion
In this paper, we present an unsupervised feature embedding method by exploring instance relations (EIR), which includes intra-instance multi-view relation and inter-instance interpolation relation. We explore intra-instance multi-view relation by aligning the distance distribution between two augmented samples and each sample in the training set. We then explore inter-instance interpolation relation by transferring the ratio of information for image sample interpolation from pixel space to feature space. Our experimental results show that our method achieves state-of-the-art or comparable performance on image classification and retrieval. In the future, more attention will be paid to relate the different instances of the same semantic category for unsupervised learning.