{'fraction_non_alphanumeric': 0.03874626865671642, 'fraction_numerical': 0.02417910447761194, 'mean_word_length': 4.969707769066287, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
TriBYOL: Triplet BYOL for Self-Supervised Representation Learning

This paper proposes a novel self-supervised learning method for learning better representations with small batch sizes. Many self-supervised learning methods based on certain forms of the siamese network have emerged and received significant attention. However, these methods need to use large batch sizes to learn good representations and require heavy computational resources. We present a new triplet network combined with a triple-view loss to improve the performance of self-supervised representation learning with small batch sizes. Experimental results show that our method can drastically outperform state-of-the-art self-supervised learning methods on several datasets in small-batch cases. Our method provides a feasible solution for self-supervised learning with real-world high-resolution images that uses small batch sizes.


INTRODUCTION
Deep supervised learning has shown outstanding performance in many areas, especially on various computer vision tasks such as image classification, object detection and semantic segmentation [1]. However, supervised learning methods have over-reliance on manually designed labels and suffer from generalization problems, and hence are meeting their bottlenecks [2]. As an alternative, selfsupervised learning is a learning framework that conforms to human cognition, which can learn information from the data itself without the need for manually designed labels [3]. Self-supervised learning has shown performance comparable to supervised learning methods on multiple tasks and has received widespread attention.
Early self-supervised learning methods often learn representations via a pretext task that applies a transformation to the input image and requires the learner to predict the properties of transformation (e.g., rotation [4] and jigsaw [5]) from the transformed image [3]. Although such transformations are beneficial for predicting 3D correlations, it is undesirable for most semantic recognition tasks. PIRL [6] proposes to learn invariant representations rather than covariant ones. The method constructs image representations similar to the representations of transformed versions of the same image and different from the representations of other images. By combining the jigsaw or rotation pretext task with PIRL, the performance surpassed supervised learning in some computer vision tasks.
Recently, self-supervised learning methods based on the siamese network [7] achieved high representation learning performance on natural images. These methods define the inputs as two augmented views from one image, then input to the siamese network and maximize the similarity between the representations of views [8]. When the batch size decreases, these methods face accuracy degradation problems due to the reduced mutual information and transformationinvariant representation [9]. Therefore, these methods typically need to increase the number of samples in a batch and need heavy computational resources (e.g., at least 4 GPUs or 32 TPU cores) for learning better representations from images [10]. For example, the stateof-the-art methods, SimCLR [11], SimSiam [12], and BYOL [13] all use batch sizes over 128. Requiring such a huge batch size is expensive and impractical in the real-world (e.g., high-resolution medical images [14][15][16] and remote sensing images [17]). Hence, it is crucial to explore self-supervised learning in small-batch cases.
In this paper, we propose a novel self-supervised learning method called TriBYOL for learning better representations with small batch sizes. Considering the appropriate addition of augmented views can increase mutual information and encourage a more transformation-invariant representation in small-batch cases [18], we present a new triplet network combined with a triple-view loss based on the state-of-the-art method BYOL [13]. The conventional triplet network [19] is a variant of the siamese network and typically has three inputs, including anchor, positive and negative samples. Different from the conventional triplet network, our method only has three augmented views from one image. Experimental results show that our method can drastically outperform the state-of-the-art self-supervised learning methods on several datasets in small-batch cases.
Our main contributions can be summarized as follows.
• We propose a novel self-supervised learning method called TriBYOL for learning better representations with small batch sizes.
• We confirm that our method can drastically outperform state-of-the-art self-supervised learning methods on several datasets in small-batch cases.

Description of TriBYOL
The overview of our method is shown in Fig. 1. Motivated by the fact that appropriate addition of augmented views can increase mutual information and encourage a more transformation-invariant representation in small-batch cases [18], different from BYOL [13] which uses the siamese network, we propose the triplet network combined with a triple-view loss for learning better representations with small batch sizes. Our method uses a triplet network to learn discriminative representations from images. The triplet network contains one online network and two target networks with the same structure, where the weights of the target networks are an exponential moving average (EMA) of the weights of the online network [20]. The conventional triplet network is a variant of the siamese network and typically has three inputs, including anchor, positive and negative samples. Different from the conventional triplet network, our method only has three augmented views from one image. Encoder E θ 1 and predictor G θ 1 belong to the online network. Encoders E θ 2 and E θ 3 belong to the target networks. Given an input image X from a dataset D without label information, three transformations t1, t2, and t3 are randomly sampled from a distribution T to generate three views V1 = t1(X), V2 = t2(X), and V3 = t3(X). Similar to BYOL, our method exchanges views for learning better representations. Y1, Y2, and Y3 are representations processed by the encoders. Z1, Z2, and Z3 are representations processed by the projector. Q1 is representations processed by the predictor. Note that we only use the predictor in the online network to make the triplet network asymmetric, which can prevent learning from collapsing, thereby improving the representation learning performance [13]. Finally, we define the loss L to compare the normalized representations from different views of the same image. The triple-view loss L comparing representations of V1, V2, and V3 is defined as follows: whereQi = Qi/||Qi||2 andẐi = Zi/||Zi||2 denote the normalized representations of Vi (i = 1, 2, 3). The consistency among views from the triplet network helps to learn discriminative representations from images. The weights of the online network (θ1) are updated as follows: Load a minibatch X from D with B samples 3: Sample three transformations from T as t1, t2, and t3 4: Generate views with random augmentation: V1 = t1(X), V2 = t2(X), and V3 = t3(X)

5:
Calculate the triple-view loss L with Eqs. (1) and (2) 6: Update the weights of the online network (θ1) with Eq. (3) 7: Update the weights of the target networks (θ2 and θ3) with Eqs. (4) and (5)  8: end for where Opt and α denote the optimizer and the learning rate, respectively. Since the stop-gradient (SG) is critical for preventing the collapse of self-supervised learning [12], the target networks (E θ 2 and E θ 3 ) are not updated using backpropagation. The weights of target networks (θ2 and θ3) are an exponential moving average of the weights of θ1 and are updated as follows: where τ denotes a momentum coefficient, and we alternately update the weights θ2 and θ3 after every iteration. After the self-supervised learning, we can use the trained encoder (E θ 1 ) for downstream tasks such as linear evaluation and fine-tuning. The self-supervised learning process of our method is summarized in Algorithm 1.

Self-supervised learning in small-batch cases
In this subsection, we propose a new problem setting and discuss its feasibility. Self-supervised learning methods based on the siamese network have achieved high representation learning performance on natural images. However, these methods typically need to increase the number of samples in a batch for learning better representations from images. Requiring such a huge batch size is expensive and impractical in the real world. For example, in practical applications, high-resolution medical images and remote sensing images are difficult to perform self-supervised learning with large batch sizes due to the limitation of GPU memory. To solve the above problem, we propose the triplet network combined with a triple-view loss based on the state-of-the-art method BYOL [13]. The appropriate addition of augmented views can increase mutual information and encourage a more transformation-invariant representation in small-batch cases [18]. Our method can provide a feasible solution for subsequent practical applications (e.g., high-resolution medical images and remote sensing images) of self-supervised learning in smallbatch cases.

Implementation details
Image Augmentations Three random views are generated by a combination of standard data augmentation methods, including cropping, resizing, flipping, color jittering, grayscaling, and Gaussian blurring based on BYOL [13] data augmentation pipeline. The following operations are performed sequentially to produce each view.
• Random cropping with an area uniformly sampled with a scale between 0.2 to 1.0, followed by resizing to a resolution of 96 × 96 pixels.
• Random horizontal flipping with an applying probability of 0.5.
• Color jittering of brightness, contrast, saturation, and hue with the strength of 0.8, 0.8, 0.8, and 0.2 with an applying probability of 0.8.
• Grayscaling with an applying probability of 0.2.
• Gaussian blurring with kernel size 9 and std between 0.1 to 2.0.
Architecture The encoders (E θ 1 , E θ 2 , and E θ 3 ) in our method are ResNet50 [21] backbone. The projectors (G θ 1 , G θ 2 , and G θ 3 ) are multilayer perceptron (MLP) whose architecture contains a linear layer with an output size of 512, a batch normalization layer, a ReLU activation function, and a linear layer with an output size of 128. The predictor (G θ 1 ) is also an MLP with the same architecture.
Optimization The optimizer (Opt) used in our method is an SGD optimizer, whose learning rate (α), momentum, and weight decay are set to 0.03, 0.9, 0.0004, respectively. We search for learning rate, momentum, and weight decay based on 40-epoch results and then apply it for longer learning such as 80 epochs or 200 epochs. We use different small batch sizes such as 32, 64, and 128, which are friendly to typical single GPU implementations.

EXPERIMENTS
In this section, we conduct experiments to verify the effectiveness of TriBYOL. First, we show the experimental settings in section 3.1. Next, we show the linear evaluation results, fine-tuning results, and transfer learning results in sections 3.2, 3.3, and 3.4, respectively. All of our experiments were conducted by using the PyTorch framework with an NVIDIA Tesla P100 GPU that has 16G memory.

Experimental settings
In the experiments, we used 9 benchmark datasets to verify the effectiveness of TriBYOL including MNIST [22], FashionMNIST [23], KMNIST [24], USPS [25], SVHN [26], CIFAR-10 [27], CIFAR-100 [27], STL-10 [28], and Tiny ImageNet [29]. We used training images without label information for self-supervised learning on well-known MNIST, FashionMNIST, KMNIST, USPS, SVHN, CIFAR-10, and CIFAR100. The STL-10 dataset contains 100,000 unlabeled images and 13,000 labeled images from 10 classes, among which 5,000 images are for training while the remaining 8,000 images are for testing. For STL-10, we used 100,000 unlabeled images and 5,000 training images without label information for selfsupervised learning. The training set of the Tiny ImageNet dataset contains 100,000 images of 200 classes (500 images for each class). We only used the training set of Tiny ImageNet without label information for self-supervised learning and transfer learning performance test in subsection 3.4.

Linear evaluation results with different small batch sizes
In this subsection, we evaluate the performance of TriBYOL by performing linear evaluation on CIFAR-10, CIFAR-100, and STL-10 with different small batch sizes (32, 64, and 128). The training epochs of self-supervised learning are set to 80. Following the linear evaluation protocol from [11], we trained a linear classifier on top of the frozen ResNet50 backbone pretrained with TriBYOL and other comparative methods for 200 epochs and test for every 10 epochs. Since self-supervised learning methods with negative sample pairs performed poorly even with a batch size of 128, we did not conduct experiments with a batch size of 32 or 64. Table 1 shows the linear evaluation results with different small batch sizes. "b32", "b64", and "b128" denote the batch size of 32, 64, and 128, respectively. From Table 1, we can see that TriBYOL drastically outperforms all comparative methods in small-batch cases. Especially, the performance of TriBYOL even exceeds that of the pretrained model on ImageNet on CIFAR-10 and CIFAR-100 with small batch sizes. When the batch size is 32, the training of SimSiam [12] collapses (i.e., output was constant), and the test results are always a certain category, resulting in 1.00% and 10.00% on CIFAR-100 and STL-10, respectively. Linear evaluation results on CIFAR-10, CIFAR-100, and STL-10 show that TriBYOL can learn better representations with different small batch sizes than other state-of-the-art self-supervised learning methods.

Fine-tuning results with different numbers of labels
In this subsection, we evaluate the performance of TriBYOL by finetuning a linear classifier with different numbers of labels on CIFAR-10, CIFAR-100, and STL-10. The number of training epochs of selfsupervised learning was set to 80. We used the ResNet50 backbone that was trained with a batch size of 128. We fine-tuned the linear classifier for 10 epochs and tested it for every epoch. Table 2 shows the fine-tuning results with different numbers of labels. "1%", "10%", and "100%" denote the percentage of used labels. We can see that TriBYOL outperforms all comparative methods with different numbers of labels. Compared with training from scratch, TriBYOL shows better representation learning ability. Finetuning results on CIFAR-10, CIFAR-100, and STL-10 show that TriBYOL can learn better representations with different numbers of labels than other state-of-the-art self-supervised learning methods.

Transfer learning results with different datasets
In this subsection, we evaluate the performance of TriBYOL by training a linear classifier on top of the frozen representations learned from Tiny ImageNet on a variety of datasets containing MNIST, FashionMNIST, KMNIST, USPS, SVHN, CIFAR-10, CIFAR-100, and STL-10. The number of training epochs of self-supervised learning was set to 80. We use the ResNet50 backbone that was trained with a batch size of 128. We trained the linear classifier for 200 epochs and tested it for every 10 epochs. Table 3 shows the transfer learning results on different datasets. As shown in Table 3, TriBYOL outperforms all comparative methods on different datasets. Furthermore, when transferring to MNIST, FashionMNIST, KMNIST, and USPS, the performance of TriBYOL even exceeds that of the pretrained model on ImageNet. Transfer learning results on different datasets show that TriBYOL can learn more generalizable representations than other state-of-the-art selfsupervised learning methods.

CONCLUSION
This paper has proposed a novel self-supervised learning method called TriBYOL. The proposed method presents a new triplet network combined with a triple-view loss based on the state-of-theart method BYOL for better representation learning performance with small batch sizes. We conduct extensive experiments of different downstream tasks containing linear evaluation, fine-tuning, and transfer learning to verify the effectiveness of TriBYOL. Experimental results show that our method can drastically outperform state-of-the-art self-supervised learning methods on several datasets in small-batch cases. Although we only take experiments in smallbatch cases due to the limitations of computing resources, we believe that our method can also have good performance even with large batch sizes. Evaluation of TriBYOL on other high-resolution image datasets in the real world will be one of our future works.