Contextual Bandits with Stochastic Experts
  We consider the problem of contextual bandits with stochastic experts, which
is a variation of the traditional stochastic contextual bandit with experts
problem. In our problem setting, we assume access to a class of stochastic
experts, where each expert is a conditional distribution over the arms given a
context. We propose upper-confidence bound (UCB) algorithms for this problem,
which employ two different importance sampling based estimators for the mean
reward for each expert. Both these estimators leverage information leakage
among the experts, thus using samples collected under all the experts to
estimate the mean reward of any given expert. This leads to instance dependent
regret bounds of $\mathcal{O}\left(\lambda(\pmb{\mu})\mathcal{M}\log T/\Delta
\right)$, where $\lambda(\pmb{\mu})$ is a term that depends on the mean rewards
of the experts, $\Delta$ is the smallest gap between the mean reward of the
optimal expert and the rest, and $\mathcal{M}$ quantifies the information
leakage among the experts. We show that under some assumptions
$\lambda(\pmb{\mu})$ is typically $\mathcal{O}(\log N)$, where $N$ is the
number of experts. We implement our algorithm with stochastic experts generated
from cost-sensitive classification oracles and show superior empirical
performance on real-world datasets, when compared to other state of the art
contextual bandit algorithms.
