DROMO: Distributionally Robust Offline Model-based Policy Optimization
  We consider the problem of offline reinforcement learning with model-based
control, whose goal is to learn a dynamics model from the experience replay and
obtain a pessimism-oriented agent under the learned model. Current model-based
constraint includes explicit uncertainty penalty and implicit conservative
regularization that pushes Q-values of out-of-distribution state-action pairs
down and the in-distribution up. While the uncertainty estimation, on which the
former relies on, can be loosely calibrated for complex dynamics, the latter
performs slightly better. To extend the basic idea of regularization without
uncertainty quantification, we propose distributionally robust offline
model-based policy optimization (DROMO), which leverages the ideas in
distributionally robust optimization to penalize a broader range of
out-of-distribution state-action pairs beyond the standard empirical
out-of-distribution Q-value minimization. We theoretically show that our method
optimizes a lower bound on the ground-truth policy evaluation, and it can be
incorporated into any existing policy gradient algorithms. We also analyze the
theoretical properties of DROMO's linear and non-linear instantiations.
