Unsupervised Reinforcement Learning in Multiple Environments
  Several recent works have been dedicated to unsupervised reinforcement
learning in a single environment, in which a policy is first pre-trained with
unsupervised interactions, and then fine-tuned towards the optimal policy for
several downstream supervised tasks defined over the same environment. Along
this line, we address the problem of unsupervised reinforcement learning in a
class of multiple environments, in which the policy is pre-trained with
interactions from the whole class, and then fine-tuned for several tasks in any
environment of the class. Notably, the problem is inherently multi-objective as
we can trade off the pre-training objective between environments in many ways.
In this work, we foster an exploration strategy that is sensitive to the most
adverse cases within the class. Hence, we cast the exploration problem as the
maximization of the mean of a critical percentile of the state visitation
entropy induced by the exploration strategy over the class of environments.
Then, we present a policy gradient algorithm, $\alpha$MEPOL, to optimize the
introduced objective through mediated interactions with the class. Finally, we
empirically demonstrate the ability of the algorithm in learning to explore
challenging classes of continuous environments and we show that reinforcement
learning greatly benefits from the pre-trained exploration strategy w.r.t.
learning from scratch.
