Meta-learnt priors slow down catastrophic forgetting in neural networks
  Current training regimes for deep learning usually involve exposure to a
single task / dataset at a time. Here we start from the observation that in
this context the trained model is not given any knowledge of anything outside
its (single-task) training distribution, and has thus no way to learn
parameters (i.e., feature detectors or policies) that could be helpful to solve
other tasks, and to limit future interference with the acquired knowledge, and
thus catastrophic forgetting. Here we show that catastrophic forgetting can be
mitigated in a meta-learning context, by exposing a neural network to multiple
tasks in a sequential manner during training. Finally, we present SeqFOMAML, a
meta-learning algorithm that implements these principles, and we evaluate it on
sequential learning problems composed by Omniglot and MiniImageNet
classification tasks.
