Evaluation of Explore-Exploit Policies in Multi-result Ranking Systems
  We analyze the problem of using Explore-Exploit techniques to improve
precision in multi-result ranking systems such as web search, query
autocompletion and news recommendation. Adopting an exploration policy directly
online, without understanding its impact on the production system, may have
unwanted consequences - the system may sustain large losses, create user
dissatisfaction, or collect exploration data which does not help improve
ranking quality. An offline framework is thus necessary to let us decide what
policy and how we should apply in a production environment to ensure positive
outcome. Here, we describe such an offline framework.
  Using the framework, we study a popular exploration policy - Thompson
sampling. We show that there are different ways of implementing it in
multi-result ranking systems, each having different semantic interpretation and
leading to different results in terms of sustained click-through-rate (CTR)
loss and expected model improvement. In particular, we demonstrate that
Thompson sampling can act as an online learner optimizing CTR, which in some
cases can lead to an interesting outcome: lift in CTR during exploration. The
observation is important for production systems as it suggests that one can get
both valuable exploration data to improve ranking performance on the long run,
and at the same time increase CTR while exploration lasts.
