Instabilities of Offline RL with Pre-Trained Neural Representation
  In offline reinforcement learning (RL), we seek to utilize offline data to
evaluate (or learn) policies in scenarios where the data are collected from a
distribution that substantially differs from that of the target policy to be
evaluated. Recent theoretical advances have shown that such sample-efficient
offline RL is indeed possible provided certain strong representational
conditions hold, else there are lower bounds exhibiting exponential error
amplification (in the problem horizon) unless the data collection distribution
has only a mild distribution shift relative to the target policy. This work
studies these issues from an empirical perspective to gauge how stable offline
RL methods are. In particular, our methodology explores these ideas when using
features from pre-trained neural networks, in the hope that these
representations are powerful enough to permit sample efficient offline RL.
Through extensive experiments on a range of tasks, we see that substantial
error amplification does occur even when using such pre-trained representations
(trained on the same task itself); we find offline RL is stable only under
extremely mild distribution shift. The implications of these results, both from
a theoretical and an empirical perspective, are that successful offline RL
(where we seek to go beyond the low distribution shift regime) requires
substantially stronger conditions beyond those which suffice for successful
supervised learning.
