Selfless Sequential Learning
  Sequential learning, also called lifelong learning, studies the problem of
learning tasks in a sequence with access restricted to only the data of the
current task. In this paper we look at a scenario with fixed model capacity,
and postulate that the learning process should not be selfish, i.e. it should
account for future tasks to be added and thus leave enough capacity for them.
To achieve Selfless Sequential Learning we study different regularization
strategies and activation functions. We find that imposing sparsity at the
level of the representation (i.e.~neuron activations) is more beneficial for
sequential learning than encouraging parameter sparsity. In particular, we
propose a novel regularizer, that encourages representation sparsity by means
of neural inhibition. It results in few active neurons which in turn leaves
more free neurons to be utilized by upcoming tasks. As neural inhibition over
an entire layer can be too drastic, especially for complex tasks requiring
strong representations, our regularizer only inhibits other neurons in a local
neighbourhood, inspired by lateral inhibition processes in the brain. We
combine our novel regularizer, with state-of-the-art lifelong learning methods
that penalize changes to important previously learned parts of the network. We
show that our new regularizer leads to increased sparsity which translates in
consistent performance improvement %over alternative regularizers we studied on
diverse datasets.
