Semi-Discriminative Representation Loss for Online Continual Learning
  The use of episodic memory in continual learning has demonstrated
effectiveness for alleviating catastrophic forgetting. In recent studies,
gradient-based approaches have been developed to make more efficient use of
compact episodic memory. Such approaches refine the gradients resulting from
new samples by those from memorized samples, aiming to reduce the diversity of
gradients from different tasks. In this paper, we clarify the relation between
diversity of gradients and discriminativeness of representations, showing
shared as well as conflicting interests between Deep Metric Learning and
continual learning, thus demonstrating pros and cons of learning discriminative
representations in continual learning. Based on these findings, we propose a
simple method -- Semi-Discriminative Representation Loss (SDRL) -- for
continual learning. In comparison with state-of-the-art methods, SDRL shows
better performance with low computational cost on multiple benchmark tasks in
the setting of online continual learning.
