Robust Continual Learning through a Comprehensively Progressive Bayesian
  Neural Network
  This work proposes a comprehensively progressive Bayesian neural network for
robust continual learning of a sequence of tasks. A Bayesian neural network is
progressively pruned and grown such that there are sufficient network resources
to represent a sequence of tasks, while the network does not explode. It starts
with the contention that similar tasks should have the same number of total
network resources, to ensure fair representation of all tasks in a continual
learning scenario. Thus, as the data for new task streams in, sufficient
neurons are added to the network such that the total number of neurons in each
layer of the network, including the shared representations with previous tasks
and individual task related representation, are equal for all tasks. The
weights that are redundant at the end of training each task are also pruned
through re-initialization, in order to be efficiently utilized in the
subsequent task. Thus, the network grows progressively, but ensures effective
utilization of network resources. We refer to our proposed method as 'Robust
Continual Learning through a Comprehensively Progressive Bayesian Neural
Network (RCL-CPB)' and evaluate the proposed approach on the MNIST data set,
under three different continual learning scenarios. Further to this, we
evaluate the performance of RCL-CPB on a homogeneous sequence of tasks using
split CIFAR100 (20 tasks of 5 classes each), and a heterogeneous sequence of
tasks using MNIST, SVHN and CIFAR10 data sets. The demonstrations and the
performance results show that the proposed strategies for progressive BNN
enable robust continual learning.
