Pessimistic Model Selection for Offline Deep Reinforcement Learning
  Deep Reinforcement Learning (DRL) has demonstrated great potentials in
solving sequential decision making problems in many applications. Despite its
promising performance, practical gaps exist when deploying DRL in real-world
scenarios. One main barrier is the over-fitting issue that leads to poor
generalizability of the policy learned by DRL. In particular, for offline DRL
with observational data, model selection is a challenging task as there is no
ground truth available for performance demonstration, in contrast with the
online setting with simulated environments. In this work, we propose a
pessimistic model selection (PMS) approach for offline DRL with a theoretical
guarantee, which features a provably effective framework for finding the best
policy among a set of candidate models. Two refined approaches are also
proposed to address the potential bias of DRL model in identifying the optimal
policy. Numerical studies demonstrated the superior performance of our approach
over existing methods.
