Lifelong Learning with Sketched Structural Regularization
  Preventing catastrophic forgetting while continually learning new tasks is an
essential problem in lifelong learning. Structural regularization (SR) refers
to a family of algorithms that mitigate catastrophic forgetting by penalizing
the network for changing its "critical parameters" from previous tasks while
learning a new one. The penalty is often induced via a quadratic regularizer
defined by an \emph{importance matrix}, e.g., the (empirical) Fisher
information matrix in the Elastic Weight Consolidation framework. In practice
and due to computational constraints, most SR methods crudely approximate the
importance matrix by its diagonal. In this paper, we propose \emph{Sketched
Structural Regularization} (Sketched SR) as an alternative approach to compress
the importance matrices used for regularizing in SR methods. Specifically, we
apply \emph{linear sketching methods} to better approximate the importance
matrices in SR algorithms. We show that sketched SR: (i) is computationally
efficient and straightforward to implement, (ii) provides an approximation
error that is justified in theory, and (iii) is method oblivious by
construction and can be adapted to any method that belongs to the structural
regularization class. We show that our proposed approach consistently improves
various SR algorithms' performance on both synthetic experiments and benchmark
continual learning tasks, including permuted-MNIST and CIFAR-100.
