Editing Factual Knowledge in Language Models
  The factual knowledge acquired during pre-training and stored in the
parameters of Language Models (LMs) can be useful in downstream tasks (e.g.,
question answering or textual inference). However, some facts can be
incorrectly induced or become obsolete over time. We present KnowledgeEditor, a
method which can be used to edit this knowledge and, thus, fix 'bugs' or
unexpected predictions without the need for expensive re-training or
fine-tuning. Besides being computationally efficient, KnowledgeEditordoes not
require any modifications in LM pre-training (e.g., the use of meta-learning).
In our approach, we train a hyper-network with constrained optimization to
modify a fact without affecting the rest of the knowledge; the trained
hyper-network is then used to predict the weight update at test time. We show
KnowledgeEditor's efficacy with two popular architectures and
knowledge-intensive tasks: i) a BERT model fine-tuned for fact-checking, and
ii) a sequence-to-sequence BART model for question answering. With our method,
changing a prediction on the specific wording of a query tends to result in a
consistent change in predictions also for its paraphrases. We show that this
can be further encouraged by exploiting (e.g., automatically-generated)
paraphrases during training. Interestingly, our hyper-network can be regarded
as a 'probe' revealing which components need to be changed to manipulate
factual knowledge; our analysis shows that the updates tend to be concentrated
on a small subset of components. Source code available at
https://github.com/nicola-decao/KnowledgeEditor
