Continual learning with hypernetworks
  Artificial neural networks suffer from catastrophic forgetting when they are
sequentially trained on multiple tasks. To overcome this problem, we present a
novel approach based on task-conditioned hypernetworks, i.e., networks that
generate the weights of a target model based on task identity. Continual
learning (CL) is less difficult for this class of models thanks to a simple key
feature: instead of recalling the input-output relations of all previously seen
data, task-conditioned hypernetworks only require rehearsing task-specific
weight realizations, which can be maintained in memory using a simple
regularizer. Besides achieving state-of-the-art performance on standard CL
benchmarks, additional experiments on long task sequences reveal that
task-conditioned hypernetworks display a very large capacity to retain previous
memories. Notably, such long memory lifetimes are achieved in a compressive
regime, when the number of trainable hypernetwork weights is comparable or
smaller than target network size. We provide insight into the structure of
low-dimensional task embedding spaces (the input space of the hypernetwork) and
show that task-conditioned hypernetworks demonstrate transfer learning.
Finally, forward information transfer is further supported by empirical results
on a challenging CL benchmark based on the CIFAR-10/100 image datasets.
