Causal Curiosity: RL Agents Discovering Self-supervised Experiments for
  Causal Representation Learning
  Animals exhibit an innate ability to learn regularities of the world through
interaction. By performing experiments in their environment, they are able to
discern the causal factors of variation and infer how they affect the world's
dynamics. Inspired by this, we attempt to equip reinforcement learning agents
with the ability to perform experiments that facilitate a categorization of the
rolled-out trajectories, and to subsequently infer the causal factors of the
environment in a hierarchical manner. We introduce {\em causal curiosity}, a
novel intrinsic reward, and show that it allows our agents to learn optimal
sequences of actions and discover causal factors in the dynamics of the
environment. The learned behavior allows the agents to infer a binary quantized
representation for the ground-truth causal factors in every environment.
Additionally, we find that these experimental behaviors are semantically
meaningful (e.g., our agents learn to lift blocks to categorize them by
weight), and are learnt in a self-supervised manner with approximately 2.5
times less data than conventional supervised planners. We show that these
behaviors can be re-purposed and fine-tuned (e.g., from lifting to pushing or
other downstream tasks). Finally, we show that the knowledge of causal factor
representations aids zero-shot learning for more complex tasks. Visit
https://sites.google.com/usc.edu/causal-curiosity/home for website.
