Thompson sampling with the online bootstrap
  Thompson sampling provides a solution to bandit problems in which new
observations are allocated to arms with the posterior probability that an arm
is optimal. While sometimes easy to implement and asymptotically optimal,
Thompson sampling can be computationally demanding in large scale bandit
problems, and its performance is dependent on the model fit to the observed
data. We introduce bootstrap Thompson sampling (BTS), a heuristic method for
solving bandit problems which modifies Thompson sampling by replacing the
posterior distribution used in Thompson sampling by a bootstrap distribution.
We first explain BTS and show that the performance of BTS is competitive to
Thompson sampling in the well-studied Bernoulli bandit case. Subsequently, we
detail why BTS using the online bootstrap is more scalable than regular
Thompson sampling, and we show through simulation that BTS is more robust to a
misspecified error distribution. BTS is an appealing modification of Thompson
sampling, especially when samples from the posterior are otherwise not
available or are costly.
