Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions
In recent years, progress in NLU has been driven by benchmarks. These
benchmarks are typically collected by crowdsourcing, where annotators write
examples based on annotation instructions crafted by dataset creators. In this
work, we hypothesize that annotators pick up on patterns in the crowdsourcing
instructions, which bias them to write similar examples that are then
over-represented in the collected data. We study this form of bias, termed
instruction bias, in 14 recent NLU benchmarks, showing that instruction
examples often exhibit concrete patterns, which are propagated by crowdworkers
to the collected data. This extends previous work (Geva et al., 2019) and
raises a new concern of whether we are modeling the dataset creator's
instructions, rather than the task. Through a series of experiments, we show
that, indeed, instruction bias can lead to overestimation of model performance,
and that models struggle to generalize beyond biases originating in the
crowdsourcing instructions. We further analyze the influence of instruction
bias in terms of pattern frequency and model size, and derive concrete
recommendations for creating future NLU benchmarks.