Potential-Based Advice for Stochastic Policy Learning
  This paper augments the reward received by a reinforcement learning agent
with potential functions in order to help the agent learn (possibly stochastic)
optimal policies. We show that a potential-based reward shaping scheme is able
to preserve optimality of stochastic policies, and demonstrate that the ability
of an agent to learn an optimal policy is not affected when this scheme is
augmented to soft Q-learning. We propose a method to impart potential based
advice schemes to policy gradient algorithms. An algorithm that considers an
advantage actor-critic architecture augmented with this scheme is proposed, and
we give guarantees on its convergence. Finally, we evaluate our approach on a
puddle-jump grid world with indistinguishable states, and the continuous state
and action mountain car environment from classical control. Our results
indicate that these schemes allow the agent to learn a stochastic optimal
policy faster and obtain a higher average reward.
