Fourier Policy Gradients
  We propose a new way of deriving policy gradient updates for reinforcement
learning. Our technique, based on Fourier analysis, recasts integrals that
arise with expected policy gradients as convolutions and turns them into
multiplications. The obtained analytical solutions allow us to capture the low
variance benefits of EPG in a broad range of settings. For the critic, we treat
trigonometric and radial basis functions, two function families with the
universal approximation property. The choice of policy can be almost arbitrary,
including mixtures or hybrid continuous-discrete probability distributions.
Moreover, we derive a general family of sample-based estimators for stochastic
policy gradients, which unifies existing results on sample-based approximation.
We believe that this technique has the potential to shape the next generation
of policy gradient approaches, powered by analytical results.
