Bootstrapping with Models: Confidence Intervals for Off-Policy
  Evaluation
  For an autonomous agent, executing a poor policy may be costly or even
dangerous. For such agents, it is desirable to determine confidence interval
lower bounds on the performance of any given policy without executing said
policy. Current methods for exact high confidence off-policy evaluation that
use importance sampling require a substantial amount of data to achieve a tight
lower bound. Existing model-based methods only address the problem in discrete
state spaces. Since exact bounds are intractable for many domains we trade off
strict guarantees of safety for more data-efficient approximate bounds. In this
context, we propose two bootstrapping off-policy evaluation methods which use
learned MDP transition models in order to estimate lower confidence bounds on
policy performance with limited data in both continuous and discrete state
spaces. Since direct use of a model may introduce bias, we derive a theoretical
upper bound on model bias for when the model transition function is estimated
with i.i.d. trajectories. This bound broadens our understanding of the
conditions under which model-based methods have high bias. Finally, we
empirically evaluate our proposed methods and analyze the settings in which
different bootstrapping off-policy confidence interval methods succeed and
fail.
