The K-Nearest Neighbour UCB algorithm for multi-armed bandits with
  covariates
  In this paper we propose and explore the k-Nearest Neighbour UCB algorithm
for multi-armed bandits with covariates. We focus on a setting where the
covariates are supported on a metric space of low intrinsic dimension, such as
a manifold embedded within a high dimensional ambient feature space. The
algorithm is conceptually simple and straightforward to implement. The
k-Nearest Neighbour UCB algorithm does not require prior knowledge of the
either the intrinsic dimension of the marginal distribution or the time
horizon. We prove a regret bound for the k-Nearest Neighbour UCB algorithm
which is minimax optimal up to logarithmic factors. In particular, the
algorithm automatically takes advantage of both low intrinsic dimensionality of
the marginal distribution over the covariates and low noise in the data,
expressed as a margin condition. In addition, focusing on the case of bounded
rewards, we give corresponding regret bounds for the k-Nearest Neighbour KL-UCB
algorithm, which is an analogue of the KL-UCB algorithm adapted to the setting
of multi-armed bandits with covariates. Finally, we present empirical results
which demonstrate the ability of both the k-Nearest Neighbour UCB and k-Nearest
Neighbour KL-UCB to take advantage of situations where the data is supported on
an unknown sub-manifold of a high-dimensional feature space.
