(Sequential) Importance Sampling Bandits
  This work extends existing multi-armed bandit (MAB) algorithms beyond their
original settings by leveraging advances in sequential Monte Carlo (SMC)
methods from the approximate inference community. We leverage Monte Carlo
estimation and, in particular, the flexibility of (sequential) importance
sampling to allow for accurate estimation of the statistics of interest within
the MAB problem. The MAB is a sequential allocation task where the goal is to
learn a policy that maximizes long term payoff, where only the reward of the
executed action is observed; i.e., sequential optimal decisions are made, while
simultaneously learning how the world operates. In the stochastic setting, the
reward for each action is generated from an unknown distribution. To decide the
next optimal action to take, one must compute sufficient statistics of this
unknown reward distribution, e.g., upper-confidence bounds (UCB), or
expectations in Thompson sampling. Closed-form expressions for these statistics
of interest are analytically intractable except for simple cases. By combining
SMC methods --- which estimate posterior densities and expectations in
probabilistic models that are analytically intractable --- with Bayesian
state-of-the-art MAB algorithms, we extend their applicability to complex
models: those for which sampling may be performed even if analytic computation
of summary statistics is infeasible --- nonlinear reward functions and dynamic
bandits. We combine SMC both for Thompson sampling and upper confident
bound-based (Bayes-UCB) policies, and study different bandit models: classic
Bernoulli and Gaussian distributed cases, as well as dynamic and context
dependent linear-Gaussian, logistic and categorical-softmax rewards.
