Task-Balanced Batch Normalization for Exemplar-based Class-Incremental
  Learning
  Batch Normalization (BN) is an essential layer for training neural network
models in various computer vision tasks. It has been widely used in continual
learning scenarios with little discussion, but we find that BN should be
carefully applied, particularly for the exemplar memory based class incremental
learning (CIL). We first analyze that the empirical mean and variance obtained
for normalization in a BN layer become highly biased toward the current task.
To tackle its significant problems in training and test phases, we propose
Task-Balanced Batch Normalization (TBBN). Given each mini-batch imbalanced
between the current and previous tasks, TBBN first reshapes and repeats the
batch, calculating near task-balanced mean and variance. Second, we show that
when the affine transformation parameters of BN are learned from a reshaped
feature map, they become less-biased toward the current task. Based on our
extensive CIL experiments with CIFAR-100 and ImageNet-100 datasets, we
demonstrate that our TBBN is easily applicable to most of existing
exemplar-based CIL algorithms, improving their performance by decreasing the
forgetting on the previous tasks.
