With Little Power Comes Great Responsibility
  Despite its importance to experimental design, statistical power (the
probability that, given a real effect, an experiment will reject the null
hypothesis) has largely been ignored by the NLP community. Underpowered
experiments make it more difficult to discern the difference between
statistical noise and meaningful model improvements, and increase the chances
of exaggerated findings. By meta-analyzing a set of existing NLP papers and
datasets, we characterize typical power for a variety of settings and conclude
that underpowered experiments are common in the NLP literature. In particular,
for several tasks in the popular GLUE benchmark, small test sets mean that most
attempted comparisons to state of the art models will not be adequately
powered. Similarly, based on reasonable assumptions, we find that the most
typical experimental design for human rating studies will be underpowered to
detect small model differences, of the sort that are frequently studied. For
machine translation, we find that typical test sets of 2000 sentences have
approximately 75% power to detect differences of 1 BLEU point. To improve the
situation going forward, we give an overview of best practices for power
analysis in NLP and release a series of notebooks to assist with future power
analyses.
