Dual-Teacher Class-Incremental Learning With Data-Free Generative Replay
  This paper proposes two novel knowledge transfer techniques for
class-incremental learning (CIL). First, we propose data-free generative replay
(DF-GR) to mitigate catastrophic forgetting in CIL by using synthetic samples
from a generative model. In the conventional generative replay, the generative
model is pre-trained for old data and shared in extra memory for later
incremental learning. In our proposed DF-GR, we train a generative model from
scratch without using any training data, based on the pre-trained
classification model from the past, so we curtail the cost of sharing
pre-trained generative models. Second, we introduce dual-teacher information
distillation (DT-ID) for knowledge distillation from two teachers to one
student. In CIL, we use DT-ID to learn new classes incrementally based on the
pre-trained model for old classes and another model (pre-)trained on the new
data for new classes. We implemented the proposed schemes on top of one of the
state-of-the-art CIL methods and showed the performance improvement on
CIFAR-100 and ImageNet datasets.
