Monte Carlo Gradient Estimation in Machine Learning
  This paper is a broad and accessible survey of the methods we have at our
disposal for Monte Carlo gradient estimation in machine learning and across the
statistical sciences: the problem of computing the gradient of an expectation
of a function with respect to parameters defining the distribution that is
integrated; the problem of sensitivity analysis. In machine learning research,
this gradient problem lies at the core of many learning problems, in
supervised, unsupervised and reinforcement learning. We will generally seek to
rewrite such gradients in a form that allows for Monte Carlo estimation,
allowing them to be easily and efficiently used and analysed. We explore three
strategies--the pathwise, score function, and measure-valued gradient
estimators--exploring their historical development, derivation, and underlying
assumptions. We describe their use in other fields, show how they are related
and can be combined, and expand on their possible generalisations. Wherever
Monte Carlo gradient estimators have been derived and deployed in the past,
important advances have followed. A deeper and more widely-held understanding
of this problem will lead to further advances, and it is these advances that we
wish to support.
