Evaluating the Faithfulness of Importance Measures in NLP by Recursively
  Masking Allegedly Important Tokens and Retraining
  To explain NLP models, importance measures such as attention inform which
inputs tokens are important for a prediction are popular. However, an open
question is how well these explanations accurately reflect a model's logic, a
property called faithfulness.
  To answer this question, we propose an new faithfulness benchmark called
Recursive ROAR. This works by recursively masking allegedly important tokens
and then retrain the model. The principle is, that this should result in worse
model performance compared to masking random tokens. The result is a
performance curve given a masking-ratio. Furthermore, we propose a summarizing
metric using the area-between-curves, which allows for easy comparison across
papers, models, and tasks.
  To provide a thorough review, we evaluate 4 different importance measures on
8 different datasets, using both LSTM-attention models and RoBERTa models.
  We find that the faithfulness of importance measures is both model-dependent
and task-dependent. This conclusion contradicts previous evaluations in both
computer vision and faithfulness of attention literature.
