Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement
  Learning
  Offline Reinforcement Learning (RL) aims to learn policies from previously
collected datasets without exploring the environment. Directly applying
off-policy algorithms to offline RL usually fails due to the extrapolation
error caused by the out-of-distribution (OOD) actions. Previous methods tackle
such problem by penalizing the Q-values of OOD actions or constraining the
trained policy to be close to the behavior policy. Nevertheless, such methods
typically prevent the generalization of value functions beyond the offline data
and also lack precise characterization of OOD data. In this paper, we propose
Pessimistic Bootstrapping for offline RL (PBRL), a purely uncertainty-driven
offline algorithm without explicit policy constraints. Specifically, PBRL
conducts uncertainty quantification via the disagreement of bootstrapped
Q-functions, and performs pessimistic updates by penalizing the value function
based on the estimated uncertainty. To tackle the extrapolating error, we
further propose a novel OOD sampling method. We show that such OOD sampling and
pessimistic bootstrapping yields provable uncertainty quantifier in linear
MDPs, thus providing the theoretical underpinning for PBRL. Extensive
experiments on D4RL benchmark show that PBRL has better performance compared to
the state-of-the-art algorithms.
