Diffusion Asymptotics for Sequential Experiments
  We propose a new diffusion-asymptotic analysis for sequentially randomized
experiments, including those that arise in solving multi-armed bandit problems.
In an experiment with $ n $ time steps, we let the mean reward gaps between
actions scale to the order $1/\sqrt{n}$ so as to preserve the difficulty of the
learning task as $n$ grows. In this regime, we show that the behavior of a
class of sequentially randomized Markov experiments converges to a diffusion
limit, given as the solution of a stochastic differential equation. The
diffusion limit thus enables us to derive refined, instance-specific
characterization of the stochastic dynamics of adaptive experiments. As an
application of this framework, we use the diffusion limit to obtain several new
insights on the regret and belief evolution of Thompson sampling. We show that
a version of Thompson sampling with an asymptotically uninformative prior
variance achieves nearly-optimal instance-specific regret scaling when the
reward gaps are relatively large. We also demonstrate that, in this regime, the
posterior beliefs underlying Thompson sampling are highly unstable over time.
