Latent-Variable Advantage-Weighted Policy Optimization for Offline RL
  Offline reinforcement learning methods hold the promise of learning policies
from pre-collected datasets without the need to query the environment for new
transitions. This setting is particularly well-suited for continuous control
robotic applications for which online data collection based on trial-and-error
is costly and potentially unsafe. In practice, offline datasets are often
heterogeneous, i.e., collected in a variety of scenarios, such as data from
several human demonstrators or from policies that act with different purposes.
Unfortunately, such datasets can exacerbate the distribution shift between the
behavior policy underlying the data and the optimal policy to be learned,
leading to poor performance. To address this challenge, we propose to leverage
latent-variable policies that can represent a broader class of policy
distributions, leading to better adherence to the training data distribution
while maximizing reward via a policy over the latent variable. As we
empirically show on a range of simulated locomotion, navigation, and
manipulation tasks, our method referred to as latent-variable
advantage-weighted policy optimization (LAPO), improves the average performance
of the next best-performing offline reinforcement learning methods by 49% on
heterogeneous datasets, and by 8% on datasets with narrow and biased
distributions.
