Pessimistic Model-based Offline Reinforcement Learning under Partial
  Coverage
  We study model-based offline Reinforcement Learning with general function
approximation without a full coverage assumption on the offline data
distribution. We present an algorithm named Constrained Pessimistic Policy
Optimization (CPPO)which leverages a general function class and uses a
constraint over the model class to encode pessimism. Under the assumption that
the ground truth model belongs to our function class (i.e., realizability in
the function class), CPPO has a PAC guarantee with offline data only providing
partial coverage, i.e., it can learn a policy that competes against any policy
that is covered by the offline data. We then demonstrate that this algorithmic
framework can be applied to many specialized Markov Decision Processes where
additional structural assumptions can further refine the concept of partial
coverage. Two notable examples are: (1) low-rank MDP with representation
learning where the partial coverage condition is defined using a relative
condition number measured by the unknown ground truth feature representation;
(2) factored MDP where the partial coverage condition is defined using density
ratio based concentrability coefficients associated with individual factors.
