Practical Benefits of Feature Feedback Under Distribution Shift
  In attempts to develop sample-efficient algorithms, researcher have explored
myriad mechanisms for collecting and exploiting feature feedback, auxiliary
annotations provided for training (but not test) instances that highlight
salient evidence. Examples include bounding boxes around objects and salient
spans in text. Despite its intuitive appeal, feature feedback has not delivered
significant gains in practical problems as assessed on iid holdout sets.
However, recent works on counterfactually augmented data suggest an alternative
benefit of supplemental annotations: lessening sensitivity to spurious patterns
and consequently delivering gains in out-of-domain evaluations. Inspired by
these findings, we hypothesize that while the numerous existing methods for
incorporating feature feedback have delivered negligible in-sample gains, they
may nevertheless generalize better out-of-domain. In experiments addressing
sentiment analysis, we show that feature feedback methods perform significantly
better on various natural out-of-domain datasets even absent differences on
in-domain evaluation. By contrast, on natural language inference tasks,
performance remains comparable. Finally, we compare those tasks where feature
feedback does (and does not) help.
