SPeCiaL: Self-Supervised Pretraining for Continual Learning
  This paper presents SPeCiaL: a method for unsupervised pretraining of
representations tailored for continual learning. Our approach devises a
meta-learning objective that differentiates through a sequential learning
process. Specifically, we train a linear model over the representations to
match different augmented views of the same image together, each view presented
sequentially. The linear model is then evaluated on both its ability to
classify images it just saw, and also on images from previous iterations. This
gives rise to representations that favor quick knowledge retention with minimal
forgetting. We evaluate SPeCiaL in the Continual Few-Shot Learning setting, and
show that it can match or outperform other supervised pretraining approaches.
