How do Quadratic Regularizers Prevent Catastrophic Forgetting: The Role
  of Interpolation
  Catastrophic forgetting undermines the effectiveness of deep neural networks
(DNNs) in scenarios such as continual learning and lifelong learning. While
several methods have been proposed to tackle this problem, there is limited
work explaining why these methods work well. This paper has the goal of better
explaining a popularly used technique for avoiding catastrophic forgetting:
quadratic regularization. We show that quadratic regularizers prevent
forgetting of past tasks by interpolating current and previous values of model
parameters at every training iteration. Over multiple training iterations, this
interpolation operation reduces the learning rates of more important model
parameters, thereby minimizing their movement. Our analysis also reveals two
drawbacks of quadratic regularization: (a) dependence of parameter
interpolation on training hyperparameters, which often leads to training
instability and (b) assignment of lower importance to deeper layers, which are
generally the place forgetting occurs in DNNs. Via a simple modification to the
order of operations, we show these drawbacks can be easily avoided, resulting
in 6.2% higher average accuracy at 4.5% lower average forgetting. Code
available at \url{https://github.com/EkdeepSLubana/QRforgetting}
