Training Networks in Null Space of Feature Covariance for Continual
  Learning
  In the setting of continual learning, a network is trained on a sequence of
tasks, and suffers from catastrophic forgetting. To balance plasticity and
stability of network in continual learning, in this paper, we propose a novel
network training algorithm called Adam-NSCL, which sequentially optimizes
network parameters in the null space of previous tasks. We first propose two
mathematical conditions respectively for achieving network stability and
plasticity in continual learning. Based on them, the network training for
sequential tasks can be simply achieved by projecting the candidate parameter
update into the approximate null space of all previous tasks in the network
training process, where the candidate parameter update can be generated by
Adam. The approximate null space can be derived by applying singular value
decomposition to the uncentered covariance matrix of all input features of
previous tasks for each linear layer. For efficiency, the uncentered covariance
matrix can be incrementally computed after learning each task. We also
empirically verify the rationality of the approximate null space at each linear
layer. We apply our approach to training networks for continual learning on
benchmark datasets of CIFAR-100 and TinyImageNet, and the results suggest that
the proposed approach outperforms or matches the state-ot-the-art continual
learning approaches.
