Learning more skills through optimistic exploration
  Unsupervised skill learning objectives (Gregor et al., 2016, Eysenbach et
al., 2018) allow agents to learn rich repertoires of behavior in the absence of
extrinsic rewards. They work by simultaneously training a policy to produce
distinguishable latent-conditioned trajectories, and a discriminator to
evaluate distinguishability by trying to infer latents from trajectories. The
hope is for the agent to explore and master the environment by encouraging each
skill (latent) to reliably reach different states. However, an inherent
exploration problem lingers: when a novel state is actually encountered, the
discriminator will necessarily not have seen enough training data to produce
accurate and confident skill classifications, leading to low intrinsic reward
for the agent and effective penalization of the sort of exploration needed to
actually maximize the objective. To combat this inherent pessimism towards
exploration, we derive an information gain auxiliary objective that involves
training an ensemble of discriminators and rewarding the policy for their
disagreement. Our objective directly estimates the epistemic uncertainty that
comes from the discriminator not having seen enough training examples, thus
providing an intrinsic reward more tailored to the true objective compared to
pseudocount-based methods (Burda et al., 2019). We call this exploration bonus
discriminator disagreement intrinsic reward, or DISDAIN. We demonstrate
empirically that DISDAIN improves skill learning both in a tabular grid world
(Four Rooms) and the 57 games of the Atari Suite (from pixels). Thus, we
encourage researchers to treat pessimism with DISDAIN.
