Overcoming catastrophic forgetting problem by weight consolidation and
  long-term memory
  Sequential learning of multiple tasks in artificial neural networks using
gradient descent leads to catastrophic forgetting, whereby previously learned
knowledge is erased during learning of new, disjoint knowledge. Here, we
propose a new approach to sequential learning which leverages the recent
discovery of adversarial examples. We use adversarial subspaces from previous
tasks to enable learning of new tasks with less interference. We apply our
method to sequentially learning to classify digits 0, 1, 2 (task 1), 4, 5, 6,
(task 2), and 7, 8, 9 (task 3) in MNIST (disjoint MNIST task). We compare and
combine our Adversarial Direction (AD) method with the recently proposed
Elastic Weight Consolidation (EWC) method for sequential learning. We train
each task for 20 epochs, which yields good initial performance (99.24% correct
task 1 performance). After training task 2, and then task 3, both plain
gradient descent (PGD) and EWC largely forget task 1 (task 1 accuracy 32.95%
for PGD and 41.02% for EWC), while our combined approach (AD+EWC) still
achieves 94.53% correct on task 1. We obtain similar results with a much more
difficult disjoint CIFAR10 task, which to our knowledge had not been attempted
before (70.10% initial task 1 performance, 67.73% after learning tasks 2 and 3
for AD+EWC, while PGD and EWC both fall to chance level). Our results suggest
that AD+EWC can provide better sequential learning performance than either PGD
or EWC.
