High-confidence error estimates for learned value functions
  Estimating the value function for a fixed policy is a fundamental problem in
reinforcement learning. Policy evaluation algorithms---to estimate value
functions---continue to be developed, to improve convergence rates, improve
stability and handle variability, particularly for off-policy learning. To
understand the properties of these algorithms, the experimenter needs
high-confidence estimates of the accuracy of the learned value functions. For
environments with small, finite state-spaces, like chains, the true value
function can be easily computed, to compute accuracy. For large, or continuous
state-spaces, however, this is no longer feasible. In this paper, we address
the largely open problem of how to obtain these high-confidence estimates, for
general state-spaces. We provide a high-confidence bound on an empirical
estimate of the value error to the true value error. We use this bound to
design an offline sampling algorithm, which stores the required quantities to
repeatedly compute value error estimates for any learned value function. We
provide experiments investigating the number of samples required by this
offline algorithm in simple benchmark reinforcement learning domains, and
highlight that there are still many open questions to be solved for this
important problem.
