Flexible and Efficient Contextual Bandits with Heterogeneous Treatment
  Effect Oracle
  Many popular contextual bandit algorithms estimate reward models to inform
decision making. However, true rewards can contain action-independent
redundancies that are not relevant for decision making and only increase the
statistical complexity of accurate estimation. It is sufficient and more
data-efficient to estimate the simplest function that explains the reward
differences between actions, that is, the heterogeneous treatment effect,
commonly understood to be more structured and simpler than the reward.
Motivated by this observation, building on recent work on oracle-based
algorithms, we design a statistically optimal and computationally efficient
algorithm using heterogeneous treatment effect estimation oracles. Our results
provide the first universal reduction of contextual bandits to a
general-purpose heterogeneous treatment effect estimation method. We show that
our approach is more robust to model misspecification than reward estimation
methods based on squared error regression oracles. Experimentally, we show the
benefits of heterogeneous treatment effect estimation in contextual bandits
over reward estimation.
