Case-based off-policy policy evaluation using prototype learning
  Importance sampling (IS) is often used to perform off-policy policy
evaluation but is prone to several issues, especially when the behavior policy
is unknown and must be estimated from data. Significant differences between the
target and behavior policies can result in uncertain value estimates due to,
for example, high variance and non-evaluated actions. If the behavior policy is
estimated using black-box models, it can be hard to diagnose potential problems
and to determine for which inputs the policies differ in their suggested
actions and resulting values. To address this, we propose estimating the
behavior policy for IS using prototype learning. We apply this approach in the
evaluation of policies for sepsis treatment, demonstrating how the prototypes
give a condensed summary of differences between the target and behavior
policies while retaining an accuracy comparable to baseline estimators. We also
describe estimated values in terms of the prototypes to better understand which
parts of the target policies have the most impact on the estimates. Using a
simulator, we study the bias resulting from restricting models to use
prototypes.
