Exploration by Maximizing R\'enyi Entropy for Reward-Free RL Framework
  Exploration is essential for reinforcement learning (RL). To face the
challenges of exploration, we consider a reward-free RL framework that
completely separates exploration from exploitation and brings new challenges
for exploration algorithms. In the exploration phase, the agent learns an
exploratory policy by interacting with a reward-free environment and collects a
dataset of transitions by executing the policy. In the planning phase, the
agent computes a good policy for any reward function based on the dataset
without further interacting with the environment. This framework is suitable
for the meta RL setting where there are many reward functions of interest. In
the exploration phase, we propose to maximize the Renyi entropy over the
state-action space and justify this objective theoretically. The success of
using Renyi entropy as the objective results from its encouragement to explore
the hard-to-reach state-actions. We further deduce a policy gradient
formulation for this objective and design a practical exploration algorithm
that can deal with complex environments. In the planning phase, we solve for
good policies given arbitrary reward functions using a batch RL algorithm.
Empirically, we show that our exploration algorithm is effective and sample
efficient, and results in superior policies for arbitrary reward functions in
the planning phase.
