Elastic weight consolidation for better bias inoculation
  The biases present in training datasets have been shown to affect models for
sentence pair classification tasks such as natural language inference (NLI) and
fact verification. While fine-tuning models on additional data has been used to
mitigate them, a common issue is that of catastrophic forgetting of the
original training dataset. In this paper, we show that elastic weight
consolidation (EWC) allows fine-tuning of models to mitigate biases while being
less susceptible to catastrophic forgetting. In our evaluation on fact
verification and NLI stress tests, we show that fine-tuning with EWC dominates
standard fine-tuning, yielding models with lower levels of forgetting on the
original (biased) dataset for equivalent gains in accuracy on the fine-tuning
(unbiased) dataset.
