Counterfactual Learning of Stochastic Policies with Continuous Actions:
  from Models to Offline Evaluation
  Counterfactual reasoning from logged data has become increasingly important
for many applications such as web advertising or healthcare. In this paper, we
address the problem of learning stochastic policies with continuous actions
from the viewpoint of counterfactual risk minimization (CRM). While the CRM
framework is appealing and well studied for discrete actions, the continuous
action case raises new challenges about modelization, optimization, and~offline
model selection with real data which turns out to be particularly challenging.
Our paper contributes to these three aspects of the CRM estimation pipeline.
First, we introduce a modelling strategy based on a joint kernel embedding of
contexts and actions, which overcomes the shortcomings of previous
discretization approaches. Second, we empirically show that the optimization
aspect of counterfactual learning is important, and we demonstrate the benefits
of proximal point algorithms and differentiable estimators. Finally, we propose
an evaluation protocol for offline policies in real-world logged systems, which
is challenging since policies cannot be replayed on test data, and we release a
new large-scale dataset along with multiple synthetic, yet realistic,
evaluation setups.
