Continual Learning by Asymmetric Loss Approximation with Single-Side
  Overestimation
  Catastrophic forgetting is a critical challenge in training deep neural
networks. Although continual learning has been investigated as a countermeasure
to the problem, it often suffers from the requirements of additional network
components and the limited scalability to a large number of tasks. We propose a
novel approach to continual learning by approximating a true loss function
using an asymmetric quadratic function with one of its sides overestimated. Our
algorithm is motivated by the empirical observation that the network parameter
updates affect the target loss functions asymmetrically. In the proposed
continual learning framework, we estimate an asymmetric loss function for the
tasks considered in the past through a proper overestimation of its unobserved
sides in training new tasks, while deriving the accurate model parameter for
the observable sides. In contrast to existing approaches, our method is free
from the side effects and achieves the state-of-the-art accuracy that is even
close to the upper-bound performance on several challenging benchmark datasets.
