Efficient Exploration via State Marginal Matching
  Exploration is critical to a reinforcement learning agent's performance in
its given environment. Prior exploration methods are often based on using
heuristic auxiliary predictions to guide policy behavior, lacking a
mathematically-grounded objective with clear properties. In contrast, we recast
exploration as a problem of State Marginal Matching (SMM), where we aim to
learn a policy for which the state marginal distribution matches a given target
state distribution. The target distribution is a uniform distribution in most
cases, but can incorporate prior knowledge if available. In effect, SMM
amortizes the cost of learning to explore in a given environment. The SMM
objective can be viewed as a two-player, zero-sum game between a state density
model and a parametric policy, an idea that we use to build an algorithm for
optimizing the SMM objective. Using this formalism, we further demonstrate that
prior work approximately maximizes the SMM objective, offering an explanation
for the success of these methods. On both simulated and real-world tasks, we
demonstrate that agents that directly optimize the SMM objective explore faster
and adapt more quickly to new tasks as compared to prior exploration methods.
