Variance-Optimal Augmentation Logging for Counterfactual Evaluation in
  Contextual Bandits
  Methods for offline A/B testing and counterfactual learning are seeing rapid
adoption in search and recommender systems, since they allow efficient reuse of
existing log data. However, there are fundamental limits to using existing log
data alone, since the counterfactual estimators that are commonly used in these
methods can have large bias and large variance when the logging policy is very
different from the target policy being evaluated. To overcome this limitation,
we explore the question of how to design data-gathering policies that most
effectively augment an existing dataset of bandit feedback with additional
observations for both learning and evaluation. To this effect, this paper
introduces Minimum Variance Augmentation Logging (MVAL), a method for
constructing logging policies that minimize the variance of the downstream
evaluation or learning problem. We explore multiple approaches to computing
MVAL policies efficiently, and find that they can be substantially more
effective in decreasing the variance of an estimator than na\"ive approaches.
