Dropout as an Implicit Gating Mechanism For Continual Learning
  In recent years, neural networks have demonstrated an outstanding ability to
achieve complex learning tasks across various domains. However, they suffer
from the "catastrophic forgetting" problem when they face a sequence of
learning tasks, where they forget the old ones as they learn new tasks. This
problem is also highly related to the "stability-plasticity dilemma". The more
plastic the network, the easier it can learn new tasks, but the faster it also
forgets previous ones. Conversely, a stable network cannot learn new tasks as
fast as a very plastic network. However, it is more reliable to preserve the
knowledge it has learned from the previous tasks. Several solutions have been
proposed to overcome the forgetting problem by making the neural network
parameters more stable, and some of them have mentioned the significance of
dropout in continual learning. However, their relationship has not been
sufficiently studied yet. In this paper, we investigate this relationship and
show that a stable network with dropout learns a gating mechanism such that for
different tasks, different paths of the network are active. Our experiments
show that the stability achieved by this implicit gating plays a very critical
role in leading to performance comparable to or better than other involved
continual learning algorithms to overcome catastrophic forgetting.
