Multiband VAE: Latent Space Alignment for Knowledge Consolidation in
  Continual Learning
  We propose a new method for unsupervised generative continual learning
through realignment of Variational Autoencoder's latent space. Deep generative
models suffer from catastrophic forgetting in the same way as other neural
structures. Recent generative continual learning works approach this problem
and try to learn from new data without forgetting previous knowledge. However,
those methods usually focus on artificial scenarios where examples share almost
no similarity between subsequent portions of data - an assumption not realistic
in the real-life applications of continual learning. In this work, we identify
this limitation and posit the goal of generative continual learning as a
knowledge accumulation task. We solve it by continuously aligning latent
representations of new data that we call bands in additional latent space where
examples are encoded independently of their source task. In addition, we
introduce a method for controlled forgetting of past data that simplifies this
process. On top of the standard continual learning benchmarks, we propose a
novel challenging knowledge consolidation scenario and show that the proposed
approach outperforms state-of-the-art by up to twofold across all experiments
and the additional real-life evaluation. To our knowledge, Multiband VAE is the
first method to show forward and backward knowledge transfer in generative
continual learning.
