APS: Active Pretraining with Successor Features
  We introduce a new unsupervised pretraining objective for reinforcement
learning. During the unsupervised reward-free pretraining phase, the agent
maximizes mutual information between tasks and states induced by the policy.
Our key contribution is a novel lower bound of this intractable quantity. We
show that by reinterpreting and combining variational successor
features~\citep{Hansen2020Fast} with nonparametric entropy
maximization~\citep{liu2021behavior}, the intractable mutual information can be
efficiently optimized. The proposed method Active Pretraining with Successor
Feature (APS) explores the environment via nonparametric entropy maximization,
and the explored data can be efficiently leveraged to learn behavior by
variational successor features. APS addresses the limitations of existing
mutual information maximization based and entropy maximization based
unsupervised RL, and combines the best of both worlds. When evaluated on the
Atari 100k data-efficiency benchmark, our approach significantly outperforms
previous methods combining unsupervised pretraining with task-specific
finetuning.
