Predicting on the Edge: Identifying Where a Larger Model Does Better
  Much effort has been devoted to making large and more accurate models, but
relatively little has been put into understanding which examples are benefiting
from the added complexity. In this paper, we demonstrate and analyze the
surprisingly tight link between a model's predictive uncertainty on individual
examples and the likelihood that larger models will improve prediction on them.
Through extensive numerical studies on the T5 encoder-decoder architecture, we
show that large models have the largest improvement on examples where the small
model is most uncertain. On more certain examples, even those where the small
model is not particularly accurate, large models are often unable to improve at
all, and can even perform worse than the smaller model. Based on these
findings, we show that a switcher model which defers examples to a larger model
when a small model is uncertain can achieve striking improvements in
performance and resource usage. We also explore committee-based uncertainty
metrics that can be more effective but less practical.
