Targeted Forgetting and False Memory Formation in Continual Learners
  through Adversarial Backdoor Attacks
  Artificial neural networks are well-known to be susceptible to catastrophic
forgetting when continually learning from sequences of tasks. Various continual
(or "incremental") learning approaches have been proposed to avoid catastrophic
forgetting, but they are typically adversary agnostic, i.e., they do not
consider the possibility of a malicious attack. In this effort, we explore the
vulnerability of Elastic Weight Consolidation (EWC), a popular continual
learning algorithm for avoiding catastrophic forgetting. We show that an
intelligent adversary can bypass the EWC's defenses, and instead cause gradual
and deliberate forgetting by introducing small amounts of misinformation to the
model during training. We demonstrate such an adversary's ability to assume
control of the model via injection of "backdoor" attack samples on both
permuted and split benchmark variants of the MNIST dataset. Importantly, once
the model has learned the adversarial misinformation, the adversary can then
control the amount of forgetting of any task. Equivalently, the malicious actor
can create a "false memory" about any task by inserting carefully-designed
backdoor samples to any fraction of the test instances of that task. Perhaps
most damaging, we show this vulnerability to be very acute; neural network
memory can be easily compromised with the addition of backdoor samples into as
little as 1% of the training data of even a single task.
