Saliency Guided Experience Packing for Replay in Continual Learning
  Artificial learning systems aspire to mimic human intelligence by continually
learning from a stream of tasks without forgetting past knowledge. One way to
enable such learning is to store past experiences in the form of input examples
in episodic memory and replay them when learning new tasks. However,
performance of such method suffers as the size of the memory becomes smaller.
In this paper, we propose a new approach for experience replay, where we select
the past experiences by looking at the saliency maps which provide visual
explanations for the model's decision. Guided by these saliency maps, we pack
the memory with only the parts or patches of the input images important for the
model's prediction. While learning a new task, we replay these memory patches
with appropriate zero-padding to remind the model about its past decisions. We
evaluate our algorithm on diverse image classification datasets and report
better performance than the state-of-the-art approaches. With qualitative and
quantitative analyses we show that our method captures richer summary of past
experiences without any memory increase, and hence performs well with small
episodic memory.
