SpaceNet: Make Free Space For Continual Learning
  The continual learning (CL) paradigm aims to enable neural networks to learn
tasks continually in a sequential fashion. The fundamental challenge in this
learning paradigm is catastrophic forgetting previously learned tasks when the
model is optimized for a new task, especially when their data is not
accessible. Current architectural-based methods aim at alleviating the
catastrophic forgetting problem but at the expense of expanding the capacity of
the model. Regularization-based methods maintain a fixed model capacity;
however, previous studies showed the huge performance degradation of these
methods when the task identity is not available during inference (e.g. class
incremental learning scenario). In this work, we propose a novel
architectural-based method referred as SpaceNet for class incremental learning
scenario where we utilize the available fixed capacity of the model
intelligently. SpaceNet trains sparse deep neural networks from scratch in an
adaptive way that compresses the sparse connections of each task in a compact
number of neurons. The adaptive training of the sparse connections results in
sparse representations that reduce the interference between the tasks.
Experimental results show the robustness of our proposed method against
catastrophic forgetting old tasks and the efficiency of SpaceNet in utilizing
the available capacity of the model, leaving space for more tasks to be
learned. In particular, when SpaceNet is tested on the well-known benchmarks
for CL: split MNIST, split Fashion-MNIST, and CIFAR-10/100, it outperforms
regularization-based methods by a big performance gap. Moreover, it achieves
better performance than architectural-based methods without model expansion and
achieved comparable results with rehearsal-based methods, while offering a huge
memory reduction.
