Zipfian environments for Reinforcement Learning
  As humans and animals learn in the natural world, they encounter
distributions of entities, situations and events that are far from uniform.
Typically, a relatively small set of experiences are encountered frequently,
while many important experiences occur only rarely. The highly-skewed,
heavy-tailed nature of reality poses particular learning challenges that humans
and animals have met by evolving specialised memory systems. By contrast, most
popular RL environments and benchmarks involve approximately uniform variation
of properties, objects, situations or tasks. How will RL algorithms perform in
worlds (like ours) where the distribution of environment features is far less
uniform? To explore this question, we develop three complementary RL
environments where the agent's experience varies according to a Zipfian
(discrete power law) distribution. On these benchmarks, we find that standard
Deep RL architectures and algorithms acquire useful knowledge of common
situations and tasks, but fail to adequately learn about rarer ones. To
understand this failure better, we explore how different aspects of current
approaches may be adjusted to help improve performance on rare events, and show
that the RL objective function, the agent's memory system and self-supervised
learning objectives can all influence an agent's ability to learn from uncommon
experiences. Together, these results show that learning robustly from skewed
experience is a critical challenge for applying Deep RL methods beyond
simulations or laboratories, and our Zipfian environments provide a basis for
measuring future progress towards this goal.
