Understanding the Role of Training Regimes in Continual Learning
  Catastrophic forgetting affects the training of neural networks, limiting
their ability to learn multiple tasks sequentially. From the perspective of the
well established plasticity-stability dilemma, neural networks tend to be
overly plastic, lacking the stability necessary to prevent the forgetting of
previous knowledge, which means that as learning progresses, networks tend to
forget previously seen tasks. This phenomenon coined in the continual learning
literature, has attracted much attention lately, and several families of
approaches have been proposed with different degrees of success. However, there
has been limited prior work extensively analyzing the impact that different
training regimes -- learning rate, batch size, regularization method-- can have
on forgetting. In this work, we depart from the typical approach of altering
the learning algorithm to improve stability. Instead, we hypothesize that the
geometrical properties of the local minima found for each task play an
important role in the overall degree of forgetting. In particular, we study the
effect of dropout, learning rate decay, and batch size, on forming training
regimes that widen the tasks' local minima and consequently, on helping it not
to forget catastrophically. Our study provides practical insights to improve
stability via simple yet effective techniques that outperform alternative
baselines.
