Nonparametric Gaussian Mixture Models for the Multi-Armed Contextual
  Bandit
  We here adopt Bayesian nonparametric mixture models to extend multi-armed
bandits in general, and Thompson sampling in particular, to scenarios where
there is reward model uncertainty. In the stochastic multi-armed bandit, where
an agent must learn a policy that maximizes long term payoff, the reward for
the selected action is generated from an unknown distribution. Thompson
sampling is a generative and interpretable multi-armed bandit algorithm that
has been shown both to perform well in practice, and to enjoy optimality
properties for certain reward functions. Nevertheless, Thompson sampling
requires knowledge of the true reward model, for calculation of expected
rewards and sampling from its parameter posterior. In this work, we extend
Thompson sampling to complex scenarios where there is model uncertainty, by
adopting a very flexible set of reward distributions: Bayesian nonparametric
Gaussian mixture models. The generative process of Bayesian nonparametric
mixtures naturally aligns with the Bayesian modeling of multi-armed bandits:
the nonparametric model autonomously determines its complexity as new rewards
are observed for the played arms. By characterizing each arm's reward
distribution with independent nonparametric mixture models, the proposed method
sequentially learns the model that best approximates the true underlying reward
distribution, achieving successful performance in complex -- not in the
exponential family -- bandits. Our contribution is valuable for practical
scenarios, as it avoids stringent case-by-case model specifications and
hyperparameter tuning, yet attains reduced regret in diverse bandit settings.
