On Wasserstein Reinforcement Learning and the Fokker-Planck equation
  Policy gradients methods often achieve better performance when the change in
policy is limited to a small Kullback-Leibler divergence. We derive policy
gradients where the change in policy is limited to a small Wasserstein distance
(or trust region). This is done in the discrete and continuous multi-armed
bandit settings with entropy regularisation. We show that in the small steps
limit with respect to the Wasserstein distance $W_2$, policy dynamics are
governed by the Fokker-Planck (heat) equation, following the
Jordan-Kinderlehrer-Otto result. This means that policies undergo diffusion and
advection, concentrating near actions with high reward. This helps elucidate
the nature of convergence in the probability matching setup, and provides
justification for empirical practices such as Gaussian policy priors and
additive gradient noise.
