Improving Zero-shot Generalization in Offline Reinforcement Learning
  using Generalized Similarity Functions
  Reinforcement learning (RL) agents are widely used for solving complex
sequential decision making tasks, but still exhibit difficulty in generalizing
to scenarios not seen during training. While prior online approaches
demonstrated that using additional signals beyond the reward function can lead
to better generalization capabilities in RL agents, i.e. using self-supervised
learning (SSL), they struggle in the offline RL setting, i.e. learning from a
static dataset. We show that performance of online algorithms for
generalization in RL can be hindered in the offline setting due to poor
estimation of similarity between observations. We propose a new
theoretically-motivated framework called Generalized Similarity Functions
(GSF), which uses contrastive learning to train an offline RL agent to
aggregate observations based on the similarity of their expected future
behavior, where we quantify this similarity using \emph{generalized value
functions}. We show that GSF is general enough to recover existing SSL
objectives while also improving zero-shot generalization performance on a
complex offline RL benchmark, offline Procgen.
