Meta-Learning for Contextual Bandit Exploration
  We describe MELEE, a meta-learning algorithm for learning a good exploration
policy in the interactive contextual bandit setting. Here, an algorithm must
take actions based on contexts, and learn based only on a reward signal from
the action taken, thereby generating an exploration/exploitation trade-off.
MELEE addresses this trade-off by learning a good exploration strategy for
offline tasks based on synthetic data, on which it can simulate the contextual
bandit setting. Based on these simulations, MELEE uses an imitation learning
strategy to learn a good exploration policy that can then be applied to true
contextual bandit tasks at test time. We compare MELEE to seven strong baseline
contextual bandit algorithms on a set of three hundred real-world datasets, on
which it outperforms alternatives in most settings, especially when differences
in rewards are large. Finally, we demonstrate the importance of having a rich
feature representation for learning how to explore.
