Shaping Advice in Deep Reinforcement Learning
  Reinforcement learning involves agents interacting with an environment to
complete tasks. When rewards provided by the environment are sparse, agents may
not receive immediate feedback on the quality of actions that they take,
thereby affecting learning of policies. In this paper, we propose to methods to
augment the reward signal from the environment with an additional reward termed
shaping advice in both single and multi-agent reinforcement learning. The
shaping advice is specified as a difference of potential functions at
consecutive time-steps. Each potential function is a function of observations
and actions of the agents. The use of potential functions is underpinned by an
insight that the total potential when starting from any state and returning to
the same state is always equal to zero. We show through theoretical analyses
and experimental validation that the shaping advice does not distract agents
from completing tasks specified by the environment reward. Theoretically, we
prove that the convergence of policy gradients and value functions when using
shaping advice implies the convergence of these quantities in the absence of
shaping advice. We design two algorithms- Shaping Advice in Single-agent
reinforcement learning (SAS) and Shaping Advice in Multi-agent reinforcement
learning (SAM). Shaping advice in SAS and SAM needs to be specified only once
at the start of training, and can easily be provided by non-experts.
Experimentally, we evaluate SAS and SAM on two tasks in single-agent
environments and three tasks in multi-agent environments that have sparse
rewards. We observe that using shaping advice results in agents learning
policies to complete tasks faster, and obtain higher rewards than algorithms
that do not use shaping advice.
