Sample Complexity of Offline Reinforcement Learning with Deep ReLU
  Networks
  We study the statistical theory of offline reinforcement learning (RL) with
deep ReLU network function approximation. We analyze a variant of fitted-Q
iteration (FQI) algorithm under a new dynamic condition that we call Besov
dynamic closure, which encompasses the conditions from prior analyses for deep
neural network function approximation. Under Besov dynamic closure, we prove
that the FQI-type algorithm enjoys the sample complexity of
$\tilde{\mathcal{O}}\left( \kappa^{1 + d/\alpha} \cdot \epsilon^{-2 -
2d/\alpha} \right)$ where $\kappa$ is a distribution shift measure, $d$ is the
dimensionality of the state-action space, $\alpha$ is the (possibly fractional)
smoothness parameter of the underlying MDP, and $\epsilon$ is a user-specified
precision. This is an improvement over the sample complexity of
$\tilde{\mathcal{O}}\left( K \cdot \kappa^{2 + d/\alpha} \cdot \epsilon^{-2 -
d/\alpha} \right)$ in the prior result [Yang et al., 2019] where $K$ is an
algorithmic iteration number which is arbitrarily large in practice.
Importantly, our sample complexity is obtained under the new general dynamic
condition and a data-dependent structure where the latter is either ignored in
prior algorithms or improperly handled by prior analyses. This is the first
comprehensive analysis for offline RL with deep ReLU network function
approximation under a general setting.
