{'fraction_non_alphanumeric': 0.028612578012481998, 'fraction_numerical': 0.012770043206913106, 'mean_word_length': 5.046052631578948, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Continually Learning Self-Supervised Representations with Projected Functional Regularization

Recent self-supervised learning methods are able to learn high-quality image representations and are closing the gap with supervised approaches. However, these methods are unable to acquire new knowledge incrementally -- they are, in fact, mostly used only as a pre-training phase over IID data. In this work we investigate self-supervised methods in continual learning regimes without any replay mechanism. We show that naive functional regularization, also known as feature distillation, leads to lower plasticity and limits continual learning performance. Instead, we propose Projected Functional Regularization in which a separate temporal projection network ensures that the newly learned feature space preserves information of the previous one, while at the same time allowing for the learning of new features. This prevents forgetting while maintaining the plasticity of the learner. Comparison with other incremental learning approaches applied to self-supervision demonstrates that our method obtains competitive performance in different scenarios and on multiple datasets.


Introduction
Self-supervised learning aims to learn high-quality image representations without the need for human annotations. A recent set of works has shown that self-supervised learning can achieve performance close to that of supervised learning [5,10,12,23], and that learned representations transferred to downstream tasks are sometimes even superior to fully-supervised representation learning [6]. These methods learn representations that are invariant with respect to a set of data augmentations. They are typically trained with contrastive losses in which multiple views of the same image (computed by applying different data augmentations) are mapped close together, whereas representations of other images are mapped far away. However, several methods show that only encouraging similarity between views from the same image (without any explicit loss to promote the distancing of negative pairs) can also obtain excellent performance [12,23]. These methods apply various mechanisms to prevent trivial solutions, including asymmetric architectures and the use of momentum updates of the model.
Recent works on self-supervised learning have in common that they assume that all training data is available during the training process. However, in many real-world applications the learner must cope with non-stationary data in which they are exposed to tasks with varying distributions of data. Continual learning relaxes the IID assumption that underlies most learning methods and studies the design of algorithms that learn from data with shifting distributions. Naively training a learner on such data, for example by simply continuing stochastic gradient descent, leads to catastrophic forgetting [46]. A variety of approaches have been proposed including various types of regularization [1,32,37,69], data replay [7,30,53,65], pseudo replay [58,64], and growing architectures [55]. Even though there is some work on unsupervised continual learning [21,36,43,55], the vast majority of existing work is on supervised continual learning [49,51].
Earlier works on self-supervised learning was based on pretext tasks like predicting rotation [22], determining patch position [17], or solving jigsaw puzzles in images [47]. Labels for these discriminative pretext tasks can be automatically computed and allow learning of meaningful feature representations of images. Recently, researchers are adapting contrastive methods for unlabeled data and operating more at an instance-level augmentation while looking for similarity or contrastive samples [5,10,23,68]. These methods rely heavily on stochastic data augmentation to produce enough similar examples to learn representations. Negative examples are randomly sampled or not used at all [12]. The results are impressive and are competitive with many supervised methods on downstream tasks [6].
We propose an approach to continual self-supervised learning that is able to learn high-quality visual feature representations from non-IID data. The learner is exposed to a changing distribution and, while learning new features on current task data, should prevent forgetting of previously ac-quired knowledge. These representations should, at the end of training, be applicable to a wide range of downstream tasks. We focus on the more restrictive, memory-free continual learning setting in which the learner is not allowed to store any samples from previous tasks. This scenario is realistic in many scenarios where data privacy and security is fundamental and often legislatively regulated.
The main contributions of this work are twofold. First, we propose a new method, called projected functional regularization, to alleviate forgetting during unsupervised representation learning without the need for an external memory of samples from previous tasks. This technique is an extension of Learning without Forgetting (LwF) and distillation in feature space. To improve the plasticity of the method we introduce a temporal projection network that provides more freedom to learn features from the current task. Secondly, we propose a set of experiments over benchmark datasets to compare with other state-of-the-art methods and use different scenarios to evaluate the functional projection role in the context of continual self-supervised representation learning. We show that the additional projection to past tasks results in better representation learning during class incremental training sessions. Without any adjustment, evaluation on a truly class incremental scenario -with only a single class per task, where many class incremental methods cannot be directly applied -our method still prevents forgetting and is able to progressively learn new features. Furthermore, we confirm that our method is generic and the results are not restricted to a particular self-supervised learning approach. In a variety of experimental settings the transferability of the learned features to different downstream tasks is maintained, confirming that the network is incrementally learning more robust representations. The influence of the regularization strength is analyzed for different regularization methods applied to self-supervised continual learning and results clearly shows the benefit of the proposed additional projector resulting both in improved plasticity (i.e. lower intransigence) and less forgetting.

Related Work
Both self-supervised and continual learning have gathered increasing interest in recent years. We briefly review the literature on both topics before articulating our contribution which combines elements of both in the form of continual self-supervised representation learning.
Self-supervised learning. Self-supervised learning has proved useful for many applications. In order to learn representations useful for a downstream task, a self-supervised pretext task can be introduced to avoid supervision. Many pretext tasks were investigated for learning image representations, including rotation prediction [22], solving jigsaw puzzles [47], determining relative patch positions [17], pre-dicting surrogate classes [18], and image colorization [71]).
In the last few years, the gap between supervised and self-supervised learning is being closed. This is primarily due to methods based on data augmentation and contrastivelike learning in which two samples are considered either similar or different to each other. This has links to earlier contrastive methods used in metric learning [25] and some extensions using triplet losses [63]. However, in the unsupervised setting without labels, different approaches must be used for creating such pairs. In SimCLR [10], similar samples are created by augmenting an input image with a random distortion, while dissimilar ones are chosen by random. To make contrastive training more efficient, the MoCo method [11,26] uses a memory bank for learned embeddings which enables efficient sampling. This memory is kept synchronized with the rest of the network during training by using a momentum encoder. The SwAV approach uses online clustering over the embedded samples [5]. SwAV does not sample negative exemplars, however, other cluster prototypes can play the role of negative examples.
Interesting are methods without any explicit contrastive pairs. The BYOL approach proposed by [23] is based on an asymmetric network with an additional MLP predictor between two outputs of the two branches. One branch is kept "offline" and updated by a momentum encoder. Sim-Siam [12] goes even further and offers a simplified solution without a momentum encoder and moreover works well without a very large mini-batch size. BarlowTwins is another simplified solution like SimSiam which uses a loss function based on correlations between each pair in a current training mini-batch [68]. Negatives are implicitly assumed to be available in each mini-batch. No asymmetry is used by the BarlowTwins network, but a larger embedding size and bigger mini-batches are preferred in this method in comparison to SimSiam.
Continual learning. Existing continual learning methods can be broadly divided into replay-based, architecturebased, and regularization-based methods [14,44]. Replaybased methods save a small amount of data from previously seen tasks [4,9] or generate synthetic data with a generative model [62,70]. Architecture-based method activate different subsets of network parameters for different tasks by allowing model parameters to grow linearly with the number of tasks. Previous works following this strategy include DER [66], Piggyback [42], PackNet [43], DAN [54], HAT [56], Ternary Masks [45] and PathNet [21]. Regularization-based methods add an additional regularization term derived from knowledge of previous tasks to the training loss. This can be done by either regularizing the weight space (constraining important parameters) [57,60] or the functional space (constraining predictions or intermediate features) [13,19,31]. EWC [32], MAS [1], REWC [38], SI [69], and RWalk [8] constrain the importance of network parameters to prevent forgetting. Methods such as LwF [37], LwM [16] and BiC [65] instead leverage knowledge distillation to regularize features or predictions.
Our approach, called Projected Functional Regularization is a functional regularization approach. Normally, these approaches distill information at the class-prediction level between an old and new model. However, in self-supervised learning this has to be applied to the embedding output. Regularizing the embedding layer is known to undermine plasticity [19], we therefore propose an additional temporal projection network that maps between the latent spaces of current and previous model. We show that this regularization prevents forgetting while obtaining improved plasticity.

Continual representation learning.
Continual unsupervised representation learning was investigated by [52] with an approach based on variational autoencoders and a Gaussian mixture model. Still, detection of new clusters and model expansion is necessary. In a very recent paper, the authors used contrastive self-supervised learning with a memory buffer for storing exemplars [40]. They proposed the LUMP method in which images from the current task and previous tasks are combined with CutMix for continual training. One of the main differences between LUMP and our approach is that ours does not require storing any data from previous tasks.
Our contribution is fundamentally different from methods using self-supervised learning to improve the learning of a sequence of supervised tasks [24,72]. Their objective is not to learn from unlabeled data, but rather to use self-supervised learning to further enrich the feature representation. The hypothesis of these works is that, for class incremental learning scenario, the features learned via selfsupervision will be more generic than ones learned from task-bounded discrimination problems.

Continual Self-supervised Representation Learning
We begin with a discussion of self-supervised representation learning, and then describe our proposed Projected Functional Regularization (PFR) approach for continual learning of self-supervised representations without the need of any memory or replay.

Self-supervised representation learning
In recent works on self-supervised learning the aim is to learn a network f θ : X → F that maps from input space X to output feature representation space F. This network is learned on unlabeled input data x drawn from distribution D. The aim is then to exploit the learned feature representation to perform any variety of downstream tasks. As an example, for the downstream task of clas- sification on some target domain, we have training data ). Adaptation to the target domain might only optimize the weights φ while keeping θ fixed on the target data, or instead might also allow θ to be fine-tuned on the target data.
In this paper we apply the BarlowTwins [68] approach to self-supervised learning of the representation network f θ . However, the proposed method is general and can be applied to other self-supervised methods. BarlowTwins does not require explicit negative samples and achieves competitive performance while remaining computationally efficient, assuming that negatives are available in each minibatch to calculate correlation between all samples in it. The BarlowTwins architecture has two branches (see the shaded area in Fig. 1). In both branches a projector network z : F → Z is used. For the sake of notational simplicity, we do not make explicit the parameters of the network z since it is not used by downstream tasks. The parameters in the backbone and projector layer are shared between the branches.
The network is trained by minimizing an invariance and a redundancy reduction term in the loss function [68]. Here, different augmented views X A and X B of the same data samples X are taken from the set of data augmentations D * . This leads to the loss defined as: where λ is a positive constant trade-off parameter between both terms, and where C is the cross-correlation matrix computed between the representations z of all samples X A and X B in a mini-batch indexed by b: The cross-correlation matrix C has values in the range of -1.0 (worst) to 1.0 (best) correlation between the projector's outputs: The invariance term of the loss function encourages the diagonal elements to be 1. As such the learned embedding will be invariant to the applied data augmentations. At the same time, the second term (redundancy reduction) keeps the offdiagonal elements close to zero and decorrelates the outputs of non-related images.

Projected Functional Regularization
Current work on self-supervised learning considers the above scenario where the learner has access to a single, large dataset which can be revisited multiple times to learn the optimal feature extractor f θ . However, for many realworld scenarios this is an unrealistic setup and the learner will have to learn the optimal feature extractor f θ from a stream of data drawn from a distribution that varies over time.
We consider scenarios in which the learner must learn from a set of tasks, each containing data drawn from a different distribution. We consider the tasks T = {1...c} where c is the current task and the data of task t follows the distributions D t . In this case we would like to find the parameters θ of the feature extractor f θ that minimize the summed loss over all tasks up to the current one c: and L c is defined as in Eq. 1. Again, D * t refers to the set of augmented samples from D t (i.e. the data from task t). However, during continual training we only have access to the data of one task, meaning that the optimal parameters must be found using only the current data D c . Naive fine-tuning results in parameters optimal for task c, however leads to catastrophic forgetting of knowledge acquired during previous tasks.
Regularization methods are among the most successful at addressing catastrophic forgetting, especially for scenarios where storing of any data from previous tasks is prohibited (which is the objective in this article). Regularization methods can be divided into two important groups: weight regularization approaches [1,32,69], which aim to find a set of weights that is both good for the current task while incurring only a small increase in loss on previous tasks, and functional regularization methods (also known as data regularization methods) which optimize weights for new tasks while incurring only minimal changes in the network outputs on previous tasks [29,48,61].
The canonical example of functional regularization, called Learning without Forgetting (LwF), was introduced in [29] and is based on knowledge distillation [28]. It was proposed for supervised continual learning and introduces an additional loss that prevents the class predictions of previous tasks on the current data from undergoing large changes while training on the current task data. This loss cannot be directly applied to self-supervised learning since it requires class predictions. However, several continual learning works have extended this idea to feature layers by replacing the modified cross-entropy distillation loss with a distance (typically L1 or L2) which can be applied to any layer output [19,39,67]. We will refer to this as feature distillation (F D) and it leads to the following loss when training task t: where θ t−1 refers to the parameters learned after training up to task t − 1, and λ f d defines the importance of the regularization term. The regularization imposed on class predictions in the original LwF paper [29] is not very restrictive: the weights can still significantly vary as long as the final network predictions do not significantly vary. It has been observed in the literature, however, that feature distillation is very restrictive and leads to continual learning methods with low plasticity [19]. In addition, this loss directly penalizes the learning of new features since these would lead to a difference between the new and old model output To address this problem we propose Projected Functional Regularization (P F R).
We would like the network to retain previous feature representation while allowing it to learn new features learned on new tasks. These new features should not be directly penalized by regularization. To do so, we introduce a temporal projection network m : Z → Z that maps the embedding learned on the current task back to the embedding learned on the previous ones (see Figure 1). The new loss is: where S(·, ·) is a cosine similarity: New features learned in f θ t (x) do not directly result in an increased loss as long as they lie in the null-space of m.
As a consequence this loss prevents forgetting of information of previous tasks while maintaining plasticity to adapt to new tasks.

Experimental Results
Here we report on a variety of experiments performed to evaluate the performance of Projected Functional Regularization for continual self-supervised representation learning without the need of an exemplar memory and replay.

Datasets
We use the following datasets in our evaluation: • CIFAR-100: Proposed by [35], this dataset consists 100 object classes in 45,000 images for training, 5,000 for validation, and 10,000 for test with 100 classes. All images are 32×32 pixels. •

Training procedure and baseline methods
In all experiments, we train a ResNet-18 [27] using SGD with an initial learning rate of 0.06 and a weight decay of 0.0001. The network is trained with cosine annealing for the first 1500 epochs. After these epochs of cosine annealing, the learning rate is reduced by a factor of 0.8 for the both projectors (view and temporal) and 0.4 for the backbone. The data augmentation process is the same as in Bar-lowTwins (which was taken from SimCLR [10]). As a temporal projector, we use an MLP with a linear layer of 512 neurons followed by a batch normalization, Relu and a second linear layer of 256 neurons for CIFAR-100 and a linear layer with 512 neurons followed by a ReLu for TinyIma-geNet.
Downstream task classifiers are by default linear with a cross-entropy loss and are trained with Adam optimizer with a learning rate 5e-3 for CIFAR-100 and 5e-2 for Tiny-ImageNet. We use validation data to implement a patience scheme that lowers the learning rate by a factor of 0.3 up to three times while training a downstream task classifier.
In our experiments we compare with the following baseline methods: • Fine-tuning (FT): The network is trained sequentially on each task without access to previous data and with no mitigation of catastrophic forgetting.
• Single Task: We perform joint training with finetuning on all data which provides an upper bound.
• Continual Joint Training (CJ): We continually perform joint training on the entire dataset seen so far. This is provides a tighter upper bound than Single Task [3].
• Feature Distillation (FD): Knowledge distillation is used as in LwF [37] to retain representation from previous tasks. We use the L2 distance as the regularization term, as is also proposed by other methods performing knowledge distillation on feature embeddings [19,39,67].
• Elastic Weight Consolidation (EWC): We use the regularization method from [32] with a contrastive loss used to estimate the diagonal of the Fisher Information Matrix.
Note that we only compare to exemplar-free methods and exclude methods that require replay from our comparison. 1

Continual representation learning
In this experiment we evaluate all methods in the incremental representation learning setting. The most straightforward way of doing this is to use the class incremental learning setting without access to labels. Specifically, we split datasets into four equal task as done in [53]. In each task we learn a self-supervised representation and in the evaluation phase we train a linear classifier using the trained backbone encoder. In order to assess the learned representation, we use all available test data to obtain the overall task-agnostic performance evaluation 2 In Table 1 we present the results for all methods. After the final task, the upper bound CJ obtains 63.6%, while a simple fine-tuning (FT) method reaches 54.8%. This is the gap where methods using regularization can improve. Joint training on all data at once outperforms CJ by 1.8%. PFR obtains an accuracy after the final task of 59.7%, while other regularization methods FD and EWC reach 57.8% and 55.0%, respectively. In addition, we show the task-aware results on CIFAR 100 of the incrementally learned representations in Figure 2.
Here the models are the same as those used in Table 1. Note that all other results in the paper are task-agnostic (no task-ID given during inference). Here, we also use training data from future tasks to train the classifier: this allows us to also evaluate the performance of the tasks that have not been seen by the feature extractor (see above diagonal elements). We observe that all methods, including FT, incrementally improve results. Only our proposed PFR method considerably outperforms FT in this setting. It is also interesting to observe that PFR obtains positive backward transfer, since the performance on task 1 and 2 improves during the consecutive training sessions.
Learned representations. In addition to evaluating accuracy on downstream classification tasks, we compare learned representation similarity with a Centered Kernel Alignment (CKA) [33]. The results are given in Figure 3. When the task is learned and immediately evaluated, the similarity is equal to one. When we finetune the model with new data, we start experiencing representation degradation -seen in decreasing values in the columns in Figure 3, left.  Figure 2. Task-aware performance after the four consecutive tasks on CIFAR-100 for several methods. Each row reports the results after each task, and columns represent on which task the model is evaluated. The last column reports average accuracy after each trained task.  With PFR, representation forgetting progresses much more slowly. FT is the worst, having the first task similarity after the last one with CKA equal 0.69, next is FD with value 0.72, and the best is PFR with 0.82.
Many task scenario. Here we consider a challenging setting with longer sequences -i.e. with more tasks. We experimented with our PFR method, fine-tuning(FT) and feature distillation(FD) on CIFAR-100 split into 50 or 100 tasks. In the case of 100 tasks, we only have a single class per task, which is an interesting setting since there are no negative classes, forcing the network to learn representations that are discriminative at the instance level. Results are presented in Figure 4a, where accuracy over all classes is given per training session for each method. Without any mitigation of forgetting, FT cannot maintain the learned representations in longer tasks sequences, dropping closely to the level of a randomly initialized backbone (23%) in the extreme case of 100 tasks. FD is also struggling on the longer sequence of 100 tasks. Only, our method shows stable results, preventing forgetting of the learned representation and progressing steadily. Similar results can be observed for 50 tasks, however the differences are not as pronounced as for the 100 task sequence.

The influence of regularization
Each regularization method is applied differently to the self-supervised network. To assess the influence of regularization and quantify its effect, we use the forgetting and intransigence measures defined in [8]. Forgetting measures the average drop in accuracy per-task during continual learning. Intransigence describes the inability of a model to learn a new task. Formally, it is the difference between a referential model accuracy at task t -for us jointly trained self-supervised model up to task t -and the current task accuracy measured using held-out data.
In Figure 4b all methods with different λ parameter values are shown for CIFAR-100 dataset. FT is a point of reference here since it uses no regularization and maximum forgetting is expected. EWC is a weight regularization method that regularizes network weight changes using the Fisher Information Matrix. With larger lambda, forgetting is lower, but at the same time we pay the price of larger intransigence. As in the other experiments, we found that weight regularization does not obtain satisfactory results when applied to continual self-supervised learning. Feature distillation represents a better trade-off. The closer the results are to the bottom left corner, the better they are, and here PFR is the clear winner. The PFR results are consistently better than FD by some margin, which implies that the additional flexibility of the model introduced by the temporal projection network indeed leads to higher plasticity while at the same time keeping forgetting low. These results are confirmed on the larger Tiny ImageNet dataset (see Figure 4c). Here, the gap between FD and PFR is much larger, showing that projected functional regularization suffers from much lower forgetting at equal intransigence values.

Generality of the Approach
In order to verify that PFR generalizes to other selfsupervised approaches, we conducted a series of experiments with SimCLR [10], SimSiam [12], and Bar-lowTwins [68]. In Table 2 we present results of fine-tuning and PFR for a ten task scenario. PFR results in 5.4%, 8.4%, and 8.1% improvement over FT after the final task. For this longer task sequence scenario the gain of our method with respect to FT is much larger when compared with results in Table 1. Starting from the second task, the effect of projected regularization is clear. In the ten task scenario, SimSiam after the first five tasks begins to outperform Bar-lowTwins, which is reflected by the results in the Task 10 column.

Transfer to downstream tasks
To better asses the quality of the trained representation, we evaluated all methods on a series of different downstream datasets. This allows us to evaluate the transferability of the learned features during the continual training process. The results for the smaller sized (32x32 images) CIFAR-100 and SVHN datasets are in Table 1 (bottom table). We observe a similar outcome as in the source dataset evaluation. The best results use our PFR method, followed by FD, EWC, and finally FT. The results are consistent during incremental learning: the better the representation is on a source classification task, the better it is on the target (SVHN) dataset after each task.
Furthermore, we trained all the methods on a larger dataset (Tiny ImageNet). We use the same procedure as for CIFAR-100 with four tasks, but with bigger images (64 × 64) and more classes (200). The results are presented in Table 3. In this dataset our method (PFR) surpasses FD, which is followed by the modified EWC and FT.
As in CIFAR100 we evaluate our networks trained on Tiny Imagenet on different downstream datasets (Cars and Aircraft). The results are given in Table 4. For these datasets, as in CIFAR100, the accuracy of the various techniques follows the same pattern: PFR yields the best results, followed by FD, EWC, and finally FT.

Conclusions and Future directions
In this paper, we proposed a method for incremental self-supervised learning without the need for any stored examples of previous tasks. Most existing regularization methods for continual learning are applied to class predictions or logits. Such approaches applied to self-supervised representation learning result in low plasticity. To address this, we propose Projected Functional Regularization via a temporal projection network that ensures that the newly  Table 3. Accuracy on TinyImageNet split into four tasks. The learned representation is evaluated using a linear classifier over all classes after each task.

Method
Task 1 Task  and Tiny ImageNet demonstrate that our approach outperforms standard feature distillation by a considerable margin. Finally, there are several limitations and future directions we discuss here. First, due to the high computational demands, experiments have been performed on low-resolution images, and they need to be confirmed for higher resolution data. Next, our method assumes access to task boundaries and cannot be directly applied in the task-free setting (without task boundaries) [2]. We think that this can be addressed by replacing the regularization based on the model from the previous task, with a regularization model that is updated with momentum. Next, continual learning of transformer architectures has only recently started [20,50]. Self-supervised learning is a key ingredient of the transformer network training, and integrating our theory with these attention-based architectures for their continual learning is especially interesting. Finally, extending the theory with a limited replay buffer is of interest and would allow to directly report class-incremental learning results where the buffer is used to compute the classifier layer.