{'fraction_non_alphanumeric': 0.02998594408870842, 'fraction_numerical': 0.006481336873340622, 'mean_word_length': 5.035344015080113, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CULT: Continual Unsupervised Learning with Typicality-Based Environment Detection

We introduce CULT (Continual Unsupervised Representation Learning with Typicality-Based Environment Detection), a new algorithm for continual unsupervised learning with variational auto-encoders. CULT uses a simple typicality metric in the latent space of a VAE to detect distributional shifts in the environment, which is used in conjunction with generative replay and an auxiliary environmental classifier to limit catastrophic forgetting in unsupervised representation learning. In our experiments, CULT significantly outperforms baseline continual unsupervised learning approaches. Code for this paper can be found here: https://github.com/oliveradk/cult


Introduction
Deep neural networks often suffer from catastrophic forgetting -when learning a new task, their performance on previously learned tasks rapidly degrades. Continual learning is the problem of overcoming catastrophic forgetting. While much of the continual learning literature focused on supervised tasks, here we are interested in unsupervised continual learning, that is, learning a task-agnostic representations of multiple environments. When learning such task-agnostic representations, it is often useful to detect environmental shifts, which in turn requires some anomaly detection method. Variational Autencoders [8] provide one possible method. Since VAE's parameterize a likelihood function on the data, we can detect new environments (anomalies) by monitoring whether the likelihood function exceed some threshold. However, multiple studies [11], [6], [4] have found that likelihood is not a reliable metric for out-of-distribution detection -out of distribution samples will often be assigned a higher likelihood than in-distribution samples. We produce similar results, finding that a VAE trained on FashionMNIST [15] assigns higher likelihood to out of distribution MNIST samples than to in distribution FashionMNIST samples. As [4] notes, this defect of likelihood may be caused by the different between likelihood and typicality. Typical samples (those that have information value close to the expected information) are different from the maximally likely samples, and while any given sample in the typical set is less likely than the maximally likely sample, a sequence of samples from the typical set is more likely than a sequence of maximally likely samples. Inspired by [2], we use this insight for a new typicality-based environmental detection algorithm, monitoring the KL divergence between a normal distribution parameterized by the mean and standard deviation of a given batch of latent variables. This method reliably detects environmental shifts from FashionMNIST to MNIST in both directions. When coupled with generative replay [14], atypicality based anomoly detectoin provides a simple, effective solution to unsupervised continual learning with variational auto encoders.

Continual Learning
Continual learning can roughly be split into three categories: rehearsal, gradient flow, and architecture expansion. Rehearsal (or replay) methods use past examples (either stored [10], or generated [14]) to train on while learning new tasks. Gradient based methods [9], [16] avoid updating parameters were important for previous tasks. Architectural expansions dynamically allocate new model capacity for new tasks and environments [13] [5].

Continual Unsupervised Learning
While much of the continual learning literature focused on continuously learning tasks, like classification and game-playing, there is also work focused on learning representations continually on unlabeled data. CURL [12] used dynamic expansion on a VAE, allocating more capacity when the likelihood of samples falls below some threshold. VASE [2] also relies on a variational auto-encoder, but uses generative replay and latent masking to prevent catastrophic forgetting, with latent masks determined by a typicality metric on each latent. We use a similar metric to detect new environments, as described below.

Anomaly Detection with Generative Models
Intuitively, likelihood should be a useful metric for anomaly/out-of-distribution detection for probabilistic generative models. However, a mounting body of work [11], [6], [4] has shown that likelihood metrics often assign higher likelihoods to out of distribution samples. [4] hypothesizes that this failure is due to the difference between likelihood and typicality, but shows that their method is also does not promote typical points as more "in-distribution". We are aware of only one work ( [2]) that directly uses a typicality-based metric for unsupervised continual learning, but this metric was used in the context of latent masking, rather than "pure" anomaly detection for environment inference.

Formal Definition
For an i.i.d sequence x 1 , x 2 , ..., x n drawn from an alphabet X , the typical set T (n) is given by all such sequences satisfying the following inequality: such that in the limit the average information of elements in the typical set converges to the entropy of the distribution.
Using this definition, we define the "most typical" set T (n) 0 as all such sequences exactly satisfying the equality

Volume in High Dimensional Space
To get an intuitive sense for why typicality-based metrics are favorable in comparison to likelihood, we need to review on how volume operates in high dimensional space. Following the work of [1], for a neighborhood N in R d , the relative volume of N decreases as a function of d. When considering a density function π defined on R d , then as d increases, most of the mass of π will be concentrated where the density function and the volume are sufficiently large. Notably, the mass does not necessarily concentrate at the mode.

The Typical Set of the Isotropic Gaussian
This interplay between density and volume is made clear in the case of the multivariate isotropic gaussian, where the distance of the "most typical" samples to the mean is a monotonically increasing function of the dimensional of the sample space. Given x ∼ N (0 ∈ R d , I ∈ R dxd ), the most typical set is given by See A for the computation. As the example above shows, drawing samples close to the mean of a multivariate gaussian becomes less likely as the dimensional of the sample space increases. So, if our distribution is approximated by a neural network, samples which are far from the most typical set (either close to the mean or very far from the mean), should be treated as anomalies. As stated below, we approximate this typicality "distance" using KL divergence from the isotropic gaussian prior.

Problem Formalism
We assume a distribution s ∼ Cat(π 1 , ..., K) over a set of K environments S = {s 1 , s 2 , ..., s K }, which share N latent generative factors from the setZ = {z 1 , z 2 , ..., z N }. z is assumed to be standard normal, that is z ∼ N (0, 1). A dataset x s ∼ p(·|z s , s) can be synthesized from latents conditioned on the environment. Together, we have the following generative process: This setup is a simplified version of that in [2], doing away with latent masking.

Inferring the Generative Factors
We use standard variational inference [8] to infer the generative factors, maximizing the ELBO and using the re parameterization trick to efficiently compute gradients through the sampling of the normally distributed latents. This ELBO loss is given by: Note that unlike [2], we do not promote disentanglement by scaling the KL-Divergence term [7], [3].

Initializing and Inferring the Environment
To detect new environments, we use a typicality metric taken over each batch. For each batch x s B , we compute the latent features z s B . We then compute the empirical mean and standard deviation of the latent batch, use these statistics to fit a diagonal gaussian, and take the KL Divergence of this gaussian and the standard normal distribution: α = N i=1 D KL (N (μ n ,σ n )||N (0, 1)) We maintain an environment index m and a boolean learning, and define two hyperparameters λ 0 , λ 1 . If alpha > λ 1 and learning is false, then we initialize a new environment, setting learning to true and incrementing m. If learning is true and α < λ 0 , then we set learning to false. Else, we use an environment inference network q ψ(s|x B ) to determine the environment s B of the batch. The intuition here is that we want to capture spikes in the atypically metric when entering a new environment, without allocating multiple indices to the same environment. See Algorithm 1 for the psuedo-code.

Preventing Catastrophic Forgetting
We use generative replay to mitigate catastrophic, leveraging the accurate environment detector by copying and freezing the current model after an environment is initialized. On each training iterations, "hallucinated" samples are generated from the frozen model via Monte-Carlo sampling. These samples are passed through the frozen model and current model, with a loss penalizing the encoder and decoder proximity of the current model to the frozen model on the samples.

Environment Inference Network
The environment inference network is a linear layer added to the final layer of the encoder network, trained to predict the environment index. The training target on the current environment is given by the current environment index. To prevent catastrophic forgetting, the network is also trained on the hallucinated samples:

Continual Unsupervised Learning with CULT
We trained CULT on FashionMNIST to MNIST and MNIST to FashionMNIST for 10 epochs on each environment. The encoder and decoder were fully connected networks with one hidden layer of size 50, and a latent bottleneck of 16 variables. We set λ 1 = 1.4, λ 0 = 0.25. Adam was used with a learning rate of 1e-3 for the main network and 1e-4 for the environment network. The environment network also uses dropout with a dropout rate of .5.

Evaluating Catastrophic Forgetting
To evaluate catastrophic forgetting, we measure the reconstruction loss of a trained model on a dataset seen earlier in training. To test how well the model has preserved representations useful for downstream tasks, we also train classifiers with 1 hidden layer of size 50 on the latent representations.

Baseline
To compare the performance of CULT to relevant baselines, we ran FashionM-NIST to MNIST while updating the replay network at a fixed interval of 500 steps, and FashionMNIST to MNIST with no generative replay.

Reconstruction Loss
CULT dramatically outperforms baseline generative replay, more than halving reconstruction loss on FashionMNIST. In fact, the CULT model trained on FashionMNIST then MNIST has only slightly higher reconstruction loss on FashionMNIST then the CULT model trained on MNIST then FashionMNIST. A similar result holds on MNIST reconstruction loss, with the MNIST to Fash-ionMNIST CULT model achieving an only slightly higher reconstruction loss on MNIST then the FashionMNIST to MNIST CULT model (See figure 1). Classifiers trained on CULT's latent representations of FashionMNIST achieve a higher accuracy then classifiers trained on the latent representations produced by generative replay and no generative replay models. On MNIST, the CULT model trained on MNIST to FashionMNIST achieves rough parity with the other FashionMNIST to MNIST models, even out performing the no replay Fashion-MNIST to MNIST model. Figure 2 illustrates these training dynamics. Note how after the datasets switch (indicated by the blue dotted line), classification accuracy rapidly degrades for the generative replay and non generative replay models, but only slightly decreases and then stabilizes for the CULT Fashion-MNIST to MNIST model.

Anomaly Detection Metrics
Tracing the KL divergence, reconstruction loss, and atypicality training curves in figure 3 validates the hypothesis that typicality based anomaly detection methods overcome outstanding issues with likelihood based metrics. In particular, note that in the transition from FashionMNIST to MNIST (again indicated by the dotted blue line), the KL divergence and reconstruction loss decrease, while the atypicality score increases. MNIST samples are assigned a higher likelihood, but are atypical -thus using atypicality based anomaly detection is more successful than likelihood based anomaly detection.

Conclusion
We presented CULT, a simple framework for using typicality-based environmental detection to facilitate continual learning with variational autoencoders and generative replay. Potential avenues for future work include running experiments on larger models and more complex datasets, experimenting with Figure 3: Anomaly Detection Metrics different typicality metrics, and performing a more rigorous analysis of typicality in generative probabilistic models.