Efficient Continual Learning in Neural Networks with Embedding
  Regularization
  Continual learning of deep neural networks is a key requirement for scaling
them up to more complex applicative scenarios and for achieving real lifelong
learning of these architectures. Previous approaches to the problem have
considered either the progressive increase in the size of the networks, or have
tried to regularize the network behavior to equalize it with respect to
previously observed tasks. In the latter case, it is essential to understand
what type of information best represents this past behavior. Common techniques
include regularizing the past outputs, gradients, or individual weights. In
this work, we propose a new, relatively simple and efficient method to perform
continual learning by regularizing instead the network internal embeddings. To
make the approach scalable, we also propose a dynamic sampling strategy to
reduce the memory footprint of the required external storage. We show that our
method performs favorably with respect to state-of-the-art approaches in the
literature, while requiring significantly less space in memory and
computational time. In addition, inspired inspired by to recent works, we
evaluate the impact of selecting a more flexible model for the activation
functions inside the network, evaluating the impact of catastrophic forgetting
on the activation functions themselves.
