AlterSGD: Finding Flat Minima for Continual Learning by Alternative
  Training
  Deep neural networks suffer from catastrophic forgetting when learning
multiple knowledge sequentially, and a growing number of approaches have been
proposed to mitigate this problem. Some of these methods achieved considerable
performance by associating the flat local minima with forgetting mitigation in
continual learning. However, they inevitably need (1) tedious hyperparameters
tuning, and (2) additional computational cost. To alleviate these problems, in
this paper, we propose a simple yet effective optimization method, called
AlterSGD, to search for a flat minima in the loss landscape. In AlterSGD, we
conduct gradient descent and ascent alternatively when the network tends to
converge at each session of learning new knowledge. Moreover, we theoretically
prove that such a strategy can encourage the optimization to converge to a flat
minima. We verify AlterSGD on continual learning benchmark for semantic
segmentation and the empirical results show that we can significantly mitigate
the forgetting and outperform the state-of-the-art methods with a large margin
under challenging continual learning protocols.
