A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental
  Learning
  Real-world applications require the classification model to adapt to new
classes without forgetting old ones. Correspondingly, Class-Incremental
Learning (CIL) aims to train a model with limited memory size to meet this
requirement. Typical CIL methods tend to save representative exemplars from
former classes to resist forgetting, while recent works find that storing
models from history can substantially boost the performance. However, the
stored models are not counted into the memory budget, which implicitly results
in unfair comparisons. We find that when counting the model size into the total
budget and comparing methods with aligned memory size, saving models do not
consistently work, especially for the case with limited memory budgets. As a
result, we need to holistically evaluate different CIL methods at different
memory scales and simultaneously consider accuracy and memory size for
measurement. On the other hand, we dive deeply into the construction of the
memory buffer for memory efficiency. By analyzing the effect of different
layers in the network, we find that shallow and deep layers have different
characteristics in CIL. Motivated by this, we propose a simple yet effective
baseline, denoted as MEMO for Memory-efficient Expandable MOdel. MEMO extends
specialized layers based on the shared generalized representations, efficiently
extracting diverse representations with modest cost and maintaining
representative exemplars. Extensive experiments on benchmark datasets validate
MEMO's competitive performance.
