Principal Gradient Direction and Confidence Reservoir Sampling for
  Continual Learning
  Task-free online continual learning aims to alleviate catastrophic forgetting
of the learner on a non-iid data stream. Experience Replay (ER) is a SOTA
continual learning method, which is broadly used as the backbone algorithm for
other replay-based methods. However, the training strategy of ER is too simple
to take full advantage of replayed examples and its reservoir sampling strategy
is also suboptimal. In this work, we propose a general proximal gradient
framework so that ER can be viewed as a special case. We further propose two
improvements accordingly: Principal Gradient Direction (PGD) and Confidence
Reservoir Sampling (CRS). In Principal Gradient Direction, we optimize a target
gradient that not only represents the major contribution of past gradients, but
also retains the new knowledge of the current gradient. We then present
Confidence Reservoir Sampling for maintaining a more informative memory buffer
based on a margin-based metric that measures the value of stored examples.
Experiments substantiate the effectiveness of both our improvements and our new
algorithm consistently boosts the performance of MIR-replay, a SOTA ER-based
method: our algorithm increases the average accuracy up to 7.9% and reduces
forgetting up to 15.4% on four datasets.
