Continual Learning with Fully Probabilistic Models
  We present an approach for continual learning (CL) that is based on fully
probabilistic (or generative) models of machine learning. In contrast to, e.g.,
GANs that are "generative" in the sense that they can generate samples, fully
probabilistic models aim at modeling the data distribution directly.
Consequently, they provide functionalities that are highly relevant for
continual learning, such as density estimation (outlier detection) and sample
generation. As a concrete realization of generative continual learning, we
propose Gaussian Mixture Replay (GMR). GMR is a pseudo-rehearsal approach using
a Gaussian Mixture Model (GMM) instance for both generator and classifier
functionalities. Relying on the MNIST, FashionMNIST and Devanagari benchmarks,
we first demonstrate unsupervised task boundary detection by GMM density
estimation, which we also use to reject untypical generated samples. In
addition, we show that GMR is capable of class-conditional sampling in the way
of a cGAN. Lastly, we verify that GMR, despite its simple structure, achieves
state-of-the-art performance on common class-incremental learning problems at
very competitive time and memory complexity.
