Learning without Forgetting
  When building a unified vision system or gradually adding new capabilities to
a system, the usual assumption is that training data for all tasks is always
available. However, as the number of tasks grows, storing and retraining on
such data becomes infeasible. A new problem arises where we add new
capabilities to a Convolutional Neural Network (CNN), but the training data for
its existing capabilities are unavailable. We propose our Learning without
Forgetting method, which uses only new task data to train the network while
preserving the original capabilities. Our method performs favorably compared to
commonly used feature extraction and fine-tuning adaption techniques and
performs similarly to multitask learning that uses original task data we assume
unavailable. A more surprising observation is that Learning without Forgetting
may be able to replace fine-tuning with similar old and new task datasets for
improved new task performance.
