Risk Bounds and Rademacher Complexity in Batch Reinforcement Learning
  This paper considers batch Reinforcement Learning (RL) with general value
function approximation. Our study investigates the minimal assumptions to
reliably estimate/minimize Bellman error, and characterizes the generalization
performance by (local) Rademacher complexities of general function classes,
which makes initial steps in bridging the gap between statistical learning
theory and batch RL. Concretely, we view the Bellman error as a surrogate loss
for the optimality gap, and prove the followings: (1) In double sampling
regime, the excess risk of Empirical Risk Minimizer (ERM) is bounded by the
Rademacher complexity of the function class. (2) In the single sampling regime,
sample-efficient risk minimization is not possible without further assumptions,
regardless of algorithms. However, with completeness assumptions, the excess
risk of FQI and a minimax style algorithm can be again bounded by the
Rademacher complexity of the corresponding function classes. (3) Fast
statistical rates can be achieved by using tools of local Rademacher
complexity. Our analysis covers a wide range of function classes, including
finite classes, linear spaces, kernel spaces, sparse linear features, etc.
