Towards Robust Bisimulation Metric Learning
  Learned representations in deep reinforcement learning (DRL) have to extract
task-relevant information from complex observations, balancing between
robustness to distraction and informativeness to the policy. Such stable and
rich representations, often learned via modern function approximation
techniques, can enable practical application of the policy improvement theorem,
even in high-dimensional continuous state-action spaces. Bisimulation metrics
offer one solution to this representation learning problem, by collapsing
functionally similar states together in representation space, which promotes
invariance to noise and distractors. In this work, we generalize value function
approximation bounds for on-policy bisimulation metrics to non-optimal policies
and approximate environment dynamics. Our theoretical results help us identify
embedding pathologies that may occur in practical use. In particular, we find
that these issues stem from an underconstrained dynamics model and an unstable
dependence of the embedding norm on the reward signal in environments with
sparse rewards. Further, we propose a set of practical remedies: (i) a norm
constraint on the representation space, and (ii) an extension of prior
approaches with intrinsic rewards and latent space regularization. Finally, we
provide evidence that the resulting method is not only more robust to sparse
reward functions, but also able to solve challenging continuous control tasks
with observational distractions, where prior methods fail.
