Regularize, Expand and Compress: Multi-task based Lifelong Learning via
  NonExpansive AutoML
  Lifelong learning, the problem of continual learning where tasks arrive in
sequence, has been lately attracting more attention in the computer vision
community. The aim of lifelong learning is to develop a system that can learn
new tasks while maintaining the performance on the previously learned tasks.
However, there are two obstacles for lifelong learning of deep neural networks:
catastrophic forgetting and capacity limitation. To solve the above issues,
inspired by the recent breakthroughs in automatically learning good neural
network architectures, we develop a Multi-task based lifelong learning via
nonexpansive AutoML framework termed Regularize, Expand and Compress (REC). REC
is composed of three stages: 1) continually learns the sequential tasks without
the learned tasks' data via a newly proposed multi-task weight consolidation
(MWC) algorithm; 2) expands the network to help the lifelong learning with
potentially improved model capability and performance by network-transformation
based AutoML; 3) compresses the expanded model after learning every new task to
maintain model efficiency and performance. The proposed MWC and REC algorithms
achieve superior performance over other lifelong learning algorithms on four
different datasets.
