Enabling Continual Learning with Differentiable Hebbian Plasticity
  Continual learning is the problem of sequentially learning new tasks or
knowledge while protecting previously acquired knowledge. However, catastrophic
forgetting poses a grand challenge for neural networks performing such learning
process. Thus, neural networks that are deployed in the real world often
struggle in scenarios where the data distribution is non-stationary (concept
drift), imbalanced, or not always fully available, i.e., rare edge cases. We
propose a Differentiable Hebbian Consolidation model which is composed of a
Differentiable Hebbian Plasticity (DHP) Softmax layer that adds a rapid
learning plastic component (compressed episodic memory) to the fixed (slow
changing) parameters of the softmax output layer; enabling learned
representations to be retained for a longer timescale. We demonstrate the
flexibility of our method by integrating well-known task-specific synaptic
consolidation methods to penalize changes in the slow weights that are
important for each target task. We evaluate our approach on the Permuted MNIST,
Split MNIST and Vision Datasets Mixture benchmarks, and introduce an imbalanced
variant of Permuted MNIST -- a dataset that combines the challenges of class
imbalance and concept drift. Our proposed model requires no additional
hyperparameters and outperforms comparable baselines by reducing forgetting.
