Bayesian decision-making under misspecified priors with applications to
  meta-learning
  Thompson sampling and other Bayesian sequential decision-making algorithms
are among the most popular approaches to tackle explore/exploit trade-offs in
(contextual) bandits. The choice of prior in these algorithms offers
flexibility to encode domain knowledge but can also lead to poor performance
when misspecified. In this paper, we demonstrate that performance degrades
gracefully with misspecification. We prove that the expected reward accrued by
Thompson sampling (TS) with a misspecified prior differs by at most
$\tilde{\mathcal{O}}(H^2 \epsilon)$ from TS with a well specified prior, where
$\epsilon$ is the total-variation distance between priors and $H$ is the
learning horizon. Our bound does not require the prior to have any parametric
form. For priors with bounded support, our bound is independent of the
cardinality or structure of the action space, and we show that it is tight up
to universal constants in the worst case.
  Building on our sensitivity analysis, we establish generic PAC guarantees for
algorithms in the recently studied Bayesian meta-learning setting and derive
corollaries for various families of priors. Our results generalize along two
axes: (1) they apply to a broader family of Bayesian decision-making
algorithms, including a Monte-Carlo implementation of the knowledge gradient
algorithm (KG), and (2) they apply to Bayesian POMDPs, the most general
Bayesian decision-making setting, encompassing contextual bandits as a special
case. Through numerical simulations, we illustrate how prior misspecification
and the deployment of one-step look-ahead (as in KG) can impact the convergence
of meta-learning in multi-armed and contextual bandits with structured and
correlated priors.
