Layerwise Optimization by Gradient Decomposition for Continual Learning
  Deep neural networks achieve state-of-the-art and sometimes super-human
performance across various domains. However, when learning tasks sequentially,
the networks easily forget the knowledge of previous tasks, known as
"catastrophic forgetting". To achieve the consistencies between the old tasks
and the new task, one effective solution is to modify the gradient for update.
Previous methods enforce independent gradient constraints for different tasks,
while we consider these gradients contain complex information, and propose to
leverage inter-task information by gradient decomposition. In particular, the
gradient of an old task is decomposed into a part shared by all old tasks and a
part specific to that task. The gradient for update should be close to the
gradient of the new task, consistent with the gradients shared by all old
tasks, and orthogonal to the space spanned by the gradients specific to the old
tasks. In this way, our approach encourages common knowledge consolidation
without impairing the task-specific knowledge. Furthermore, the optimization is
performed for the gradients of each layer separately rather than the
concatenation of all gradients as in previous works. This effectively avoids
the influence of the magnitude variation of the gradients in different layers.
Extensive experiments validate the effectiveness of both gradient-decomposed
optimization and layer-wise updates. Our proposed method achieves
state-of-the-art results on various benchmarks of continual learning.
