GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep
  Reinforcement Learning
  Designing agents, capable of learning autonomously a wide range of skills is
critical in order to increase the scope of reinforcement learning. It will both
increase the diversity of learned skills and reduce the burden of manually
designing reward functions for each skill. Self-supervised agents, setting
their own goals, and trying to maximize the diversity of those goals have shown
great promise towards this end. However, a currently known limitation of agents
trying to maximize the diversity of sampled goals is that they tend to get
attracted to noise or more generally to parts of the environments that cannot
be controlled (distractors). When agents have access to predefined goal
features or expert knowledge, absolute Learning Progress (ALP) provides a way
to distinguish between regions that can be controlled and those that cannot.
However, those methods often fall short when the agents are only provided with
raw sensory inputs such as images. In this work we extend those concepts to
unsupervised image-based goal exploration. We propose a framework that allows
agents to autonomously identify and ignore noisy distracting regions while
searching for novelty in the learnable regions to both improve overall
performance and avoid catastrophic forgetting. Our framework can be combined
with any state-of-the-art novelty seeking goal exploration approaches. We
construct a rich 3D image based environment with distractors. Experiments on
this environment show that agents using our framework successfully identify
interesting regions of the environment, resulting in drastically improved
performances. The source code is available at
https://sites.google.com/view/grimgep.
