GROWN: GRow Only When Necessary for Continual Learning
  Catastrophic forgetting is a notorious issue in deep learning, referring to
the fact that Deep Neural Networks (DNN) could forget the knowledge about
earlier tasks when learning new tasks. To address this issue, continual
learning has been developed to learn new tasks sequentially and perform
knowledge transfer from the old tasks to the new ones without forgetting. While
recent structure-based learning methods show the capability of alleviating the
forgetting problem, these methods start from a redundant full-size network and
require a complex learning process to gradually grow-and-prune or search the
network structure for each task, which is inefficient. To address this problem
and enable efficient network expansion for new tasks, we first develop a
learnable sparse growth method eliminating the additional pruning/searching
step in previous structure-based methods. Building on this learnable sparse
growth method, we then propose GROWN, a novel end-to-end continual learning
framework to dynamically grow the model only when necessary. Different from all
previous structure-based methods, GROWN starts from a small seed network,
instead of a full-sized one. We validate GROWN on multiple datasets against
state-of-the-art methods, which shows superior performance in both accuracy and
model size. For example, we achieve 1.0\% accuracy gain on average compared to
the current SOTA results on CIFAR-100 Superclass 20 tasks setting.
