A survey of Monte Carlo methods for noisy and costly densities with
  application to reinforcement learning
  This survey gives an overview of Monte Carlo methodologies using surrogate
models, for dealing with densities which are intractable, costly, and/or noisy.
This type of problem can be found in numerous real-world scenarios, including
stochastic optimization and reinforcement learning, where each evaluation of a
density function may incur some computationally-expensive or even physical
(real-world activity) cost, likely to give different results each time. The
surrogate model does not incur this cost, but there are important trade-offs
and considerations involved in the choice and design of such methodologies. We
classify the different methodologies into three main classes and describe
specific instances of algorithms under a unified notation. A modular scheme
which encompasses the considered methods is also presented. A range of
application scenarios is discussed, with special attention to the
likelihood-free setting and reinforcement learning. Several numerical
comparisons are also provided.
