Learning Curves for Continual Learning in Neural Networks:
  Self-Knowledge Transfer and Forgetting
  Sequential training from task to task is becoming one of the major objects in
deep learning applications such as continual learning and transfer learning.
Nevertheless, it remains unclear under what conditions the trained model's
performance improves or deteriorates. To deepen our understanding of sequential
training, this study provides a theoretical analysis of generalization
performance in a solvable case of continual learning. We consider neural
networks in the neural tangent kernel (NTK) regime that continually learn
target functions from task to task, and investigate the generalization by using
an established statistical mechanical analysis of kernel ridge-less regression.
We first show characteristic transitions from positive to negative transfer.
More similar targets above a specific critical value can achieve positive
knowledge transfer for the subsequent task while catastrophic forgetting occurs
even with very similar targets. Next, we investigate a variant of continual
learning which supposes the same target function in multiple tasks. Even for
the same target, the trained model shows some transfer and forgetting depending
on the sample size of each task. We can guarantee that the generalization error
monotonically decreases from task to task for equal sample sizes while
unbalanced sample sizes deteriorate the generalization. We respectively refer
to these improvement and deterioration as self-knowledge transfer and
forgetting, and empirically confirm them in realistic training of deep neural
networks as well.
