Automatically Extracting Challenge Sets for Non local Phenomena in
  Neural Machine Translation
  We show that the state of the art Transformer Machine Translation (MT) model
is not biased towards monotonic reordering (unlike previous recurrent neural
network models), but that nevertheless, long-distance dependencies remain a
challenge for the model. Since most dependencies are short-distance, common
evaluation metrics will be little influenced by how well systems perform on
them. We, therefore, propose an automatic approach for extracting challenge
sets replete with long-distance dependencies and argue that evaluation using
this methodology provides a complementary perspective on system performance. To
support our claim, we compile challenge sets for English-German and
German-English, which are much larger than any previously released challenge
set for MT. The extracted sets are large enough to allow reliable automatic
evaluation, which makes the proposed approach a scalable and practical solution
for evaluating MT performance on the long-tail of syntactic phenomena.
