RvS: What is Essential for Offline RL via Supervised Learning?
  Recent work has shown that supervised learning alone, without temporal
difference (TD) learning, can be remarkably effective for offline RL. When does
this hold true, and which algorithmic components are necessary? Through
extensive experiments, we boil supervised learning for offline RL down to its
essential elements. In every environment suite we consider, simply maximizing
likelihood with a two-layer feedforward MLP is competitive with
state-of-the-art results of substantially more complex methods based on TD
learning or sequence modeling with Transformers. Carefully choosing model
capacity (e.g., via regularization or architecture) and choosing which
information to condition on (e.g., goals or rewards) are critical for
performance. These insights serve as a field guide for practitioners doing
Reinforcement Learning via Supervised Learning (which we coin "RvS learning").
They also probe the limits of existing RvS methods, which are comparatively
weak on random data, and suggest a number of open problems.
