Don't Read Too Much into It: Adaptive Computation for Open-Domain
  Question Answering
  Most approaches to Open-Domain Question Answering consist of a light-weight
retriever that selects a set of candidate passages, and a computationally
expensive reader that examines the passages to identify the correct answer.
Previous works have shown that as the number of retrieved passages increases,
so does the performance of the reader. However, they assume all retrieved
passages are of equal importance and allocate the same amount of computation to
them, leading to a substantial increase in computational cost. To reduce this
cost, we propose the use of adaptive computation to control the computational
budget allocated for the passages to be read. We first introduce a technique
operating on individual passages in isolation which relies on anytime
prediction and a per-layer estimation of an early exit probability. We then
introduce SkylineBuilder, an approach for dynamically deciding on which passage
to allocate computation at each step, based on a resource allocation policy
trained via reinforcement learning. Our results on SQuAD-Open show that
adaptive computation with global prioritisation improves over several strong
static and adaptive methods, leading to a 4.3x reduction in computation while
retaining 95% performance of the full model.
