Closed-Loop Memory GAN for Continual Learning
  Sequential learning of tasks using gradient descent leads to an unremitting
decline in the accuracy of tasks for which training data is no longer
available, termed catastrophic forgetting. Generative models have been explored
as a means to approximate the distribution of old tasks and bypass storage of
real data. Here we propose a cumulative closed-loop memory replay GAN (CloGAN)
provided with external regularization by a small memory unit selected for
maximum sample diversity. We evaluate incremental class learning using a
notoriously hard paradigm, single-headed learning, in which each task is a
disjoint subset of classes in the overall dataset, and performance is evaluated
on all previous classes. First, we show that when constructing a dynamic memory
unit to preserve sample heterogeneity, model performance asymptotically
approaches training on the full dataset. We then show that using a stochastic
generator to continuously output fresh new images during training increases
performance significantly further meanwhile generating quality images. We
compare our approach to several baselines including fine-tuning by gradient
descent (FGD), Elastic Weight Consolidation (EWC), Deep Generative Replay (DGR)
and Memory Replay GAN (MeRGAN). Our method has very low long-term memory cost,
the memory unit, as well as negligible intermediate memory storage.
