Off-Policy Fitted Q-Evaluation with Differentiable Function
  Approximators: Z-Estimation and Inference Theory
  Off-Policy Evaluation (OPE) serves as one of the cornerstones in
Reinforcement Learning (RL). Fitted Q Evaluation (FQE) with various function
approximators, especially deep neural networks, has gained practical success.
While statistical analysis has proved FQE to be minimax-optimal with tabular,
linear and several nonparametric function families, its practical performance
with more general function approximator is less theoretically understood. We
focus on FQE with general differentiable function approximators, making our
theory applicable to neural function approximations. We approach this problem
using the Z-estimation theory and establish the following results: The FQE
estimation error is asymptotically normal with explicit variance determined
jointly by the tangent space of the function class at the ground truth, the
reward structure, and the distribution shift due to off-policy learning; The
finite-sample FQE error bound is dominated by the same variance term, and it
can also be bounded by function class-dependent divergence, which measures how
the off-policy distribution shift intertwines with the function approximator.
In addition, we study bootstrapping FQE estimators for error distribution
inference and estimating confidence intervals, accompanied by a Cramer-Rao
lower bound that matches our upper bounds. The Z-estimation analysis provides a
generalizable theoretical framework for studying off-policy estimation in RL
and provides sharp statistical theory for FQE with differentiable function
approximators.
