Marginal Policy Gradients: A Unified Family of Estimators for Bounded
  Action Spaces with Applications
  Many complex domains, such as robotics control and real-time strategy (RTS)
games, require an agent to learn a continuous control. In the former, an agent
learns a policy over $\mathbb{R}^d$ and in the latter, over a discrete set of
actions each of which is parametrized by a continuous parameter. Such problems
are naturally solved using policy based reinforcement learning (RL) methods,
but unfortunately these often suffer from high variance leading to instability
and slow convergence. Unnecessary variance is introduced whenever policies over
bounded action spaces are modeled using distributions with unbounded support by
applying a transformation $T$ to the sampled action before execution in the
environment. Recently, the variance reduced clipped action policy gradient
(CAPG) was introduced for actions in bounded intervals, but to date no variance
reduced methods exist when the action is a direction, something often seen in
RTS games. To this end we introduce the angular policy gradient (APG), a
stochastic policy gradient method for directional control. With the marginal
policy gradients family of estimators we present a unified analysis of the
variance reduction properties of APG and CAPG; our results provide a stronger
guarantee than existing analyses for CAPG. Experimental results on a popular
RTS game and a navigation task show that the APG estimator offers a substantial
improvement over the standard policy gradient.
