Is the Policy Gradient a Gradient?
  The policy gradient theorem describes the gradient of the expected discounted
return with respect to an agent's policy parameters. However, most policy
gradient methods drop the discount factor from the state distribution and
therefore do not optimize the discounted objective. What do they optimize
instead? This has been an open question for several years, and this lack of
theoretical clarity has lead to an abundance of misstatements in the
literature. We answer this question by proving that the update direction
approximated by most methods is not the gradient of any function. Further, we
argue that algorithms that follow this direction are not guaranteed to converge
to a "reasonable" fixed point by constructing a counterexample wherein the
fixed point is globally pessimal with respect to both the discounted and
undiscounted objectives. We motivate this work by surveying the literature and
showing that there remains a widespread misunderstanding regarding discounted
policy gradient methods, with errors present even in highly-cited papers
published at top conferences.
