SPACE: Structured Compression and Sharing of Representational Space for
  Continual Learning
  Humans learn adaptively and efficiently throughout their lives. However,
incrementally learning tasks causes artificial neural networks to overwrite
relevant information learned about older tasks, resulting in 'Catastrophic
Forgetting'. Efforts to overcome this phenomenon often utilize resources
poorly, for instance, by growing the network architecture or needing to save
parametric importance scores, or violate data privacy between tasks. To tackle
this, we propose SPACE, an algorithm that enables a network to learn
continually and efficiently by partitioning the learnt space into a Core space,
that serves as the condensed knowledge base over previously learned tasks, and
a Residual space, which is akin to a scratch space for learning the current
task. After learning each task, the Residual is analyzed for redundancy, both
within itself and with the learnt Core space. A minimal number of extra
dimensions required to explain the current task are added to the Core space and
the remaining Residual is freed up for learning the next task. We evaluate our
algorithm on P-MNIST, CIFAR and a sequence of 8 different datasets, and achieve
comparable accuracy to the state-of-the-art methods while overcoming
catastrophic forgetting. Additionally, our algorithm is well suited for
practical use. The partitioning algorithm analyzes all layers in one shot,
ensuring scalability to deeper networks. Moreover, the analysis of dimensions
translates to filter-level sparsity, and the structured nature of the resulting
architecture gives us up to 5x improvement in energy efficiency during task
inference over the current state-of-the-art.
