Contextual Bandit with Adaptive Feature Extraction
  We consider an online decision making setting known as contextual bandit
problem, and propose an approach for improving contextual bandit performance by
using an adaptive feature extraction (representation learning) based on online
clustering. Our approach starts with an off-line pre-training on unlabeled
history of contexts (which can be exploited by our approach, but not by the
standard contextual bandit), followed by an online selection and adaptation of
encoders. Specifically, given an input sample (context), the proposed approach
selects the most appropriate encoding function to extract a feature vector
which becomes an input for a contextual bandit, and updates both the bandit and
the encoding function based on the context and on the feedback (reward). Our
experiments on a variety of datasets, and both in stationary and non-stationary
environments of several kinds demonstrate clear advantages of the proposed
adaptive representation learning over the standard contextual bandit based on
"raw" input contexts.
