The Effect of Natural Distribution Shift on Question Answering Models
  We build four new test sets for the Stanford Question Answering Dataset
(SQuAD) and evaluate the ability of question-answering systems to generalize to
new data. Our first test set is from the original Wikipedia domain and measures
the extent to which existing systems overfit the original test set. Despite
several years of heavy test set re-use, we find no evidence of adaptive
overfitting. The remaining three test sets are constructed from New York Times
articles, Reddit posts, and Amazon product reviews and measure robustness to
natural distribution shifts. Across a broad range of models, we observe average
performance drops of 3.8, 14.0, and 17.4 F1 points, respectively. In contrast,
a strong human baseline matches or exceeds the performance of SQuAD models on
the original domain and exhibits little to no drop in new domains. Taken
together, our results confirm the surprising resilience of the holdout method
and emphasize the need to move towards evaluation metrics that incorporate
robustness to natural distribution shifts.
