Conditional Computation for Continual Learning
  Catastrophic forgetting of connectionist neural networks is caused by the
global sharing of parameters among all training examples. In this study, we
analyze parameter sharing under the conditional computation framework where the
parameters of a neural network are conditioned on each input example. At one
extreme, if each input example uses a disjoint set of parameters, there is no
sharing of parameters thus no catastrophic forgetting. At the other extreme, if
the parameters are the same for every example, it reduces to the conventional
neural network. We then introduce a clipped version of maxout networks which
lies in the middle, i.e. parameters are shared partially among examples. Based
on the parameter sharing analysis, we can locate a limited set of examples that
are interfered when learning a new example. We propose to perform rehearsal on
this set to prevent forgetting, which is termed as conditional rehearsal.
Finally, we demonstrate the effectiveness of the proposed method in an online
non-stationary setup, where updates are made after each new example and the
distribution of the received example shifts over time.
