Risk and optimal policies in bandit experiments
  We provide a decision theoretic analysis of bandit experiments. Working
within the framework of diffusion asymptotics, we define suitable notions of
asymptotic Bayes and minimax risk for these experiments. For normally
distributed rewards, the minimal Bayes risk can be characterized as the
solution to a nonlinear second-order partial differential equation (PDE). Using
a limit of experiments approach, we show that this PDE characterization also
holds asymptotically under both parametric and non-parametric distribution of
the rewards. The approach further describes the state variables it is
asymptotically sufficient to restrict attention to, and therefore suggests a
practical strategy for dimension reduction. The upshot is that we can
approximate the dynamic programming problem defining the bandit experiment with
a PDE which can be efficiently solved using sparse matrix routines. We derive
the optimal Bayes and minimax policies from the numerical solutions to these
PDEs. The proposed policies substantially dominate existing methods such as
Thompson sampling. The framework can be generalized to allow for time
discounting and pure exploration motives.
