Bootstrapping Upper Confidence Bound
  Upper Confidence Bound (UCB) method is arguably the most celebrated one used
in online decision making with partial information feedback. Existing
techniques for constructing confidence bounds are typically built upon various
concentration inequalities, which thus lead to over-exploration. In this paper,
we propose a non-parametric and data-dependent UCB algorithm based on the
multiplier bootstrap. To improve its finite sample performance, we further
incorporate second-order correction into the above construction. In theory, we
derive both problem-dependent and problem-independent regret bounds for
multi-armed bandits under a much weaker tail assumption than the standard
sub-Gaussianity. Numerical results demonstrate significant regret reductions by
our method, in comparison with several baselines in a range of multi-armed and
linear bandit problems.
