Model-Based Policy Search Using Monte Carlo Gradient Estimation with
  Real Systems Application
  In this paper, we present a Model-Based Reinforcement Learning algorithm
named Monte Carlo Probabilistic Inference for Learning COntrol (MC-PILCO). The
algorithm relies on Gaussian Processes (GPs) to model the system dynamics and
on a Monte Carlo approach to estimate the policy gradient. This defines a
framework in which we ablate the choice of the following components: (i) the
selection of the cost function, (ii) the optimization of policies using
dropout, (iii) an improved data efficiency through the use of structured
kernels in the GP models. The combination of the aforementioned aspects affects
dramatically the performance of MC-PILCO. Numerical comparisons in a simulated
cart-pole environment show that MC-PILCO exhibits better data-efficiency and
control performance w.r.t. state-of-the-art GP-based MBRL algorithms. Finally,
we apply MC-PILCO to real systems, considering in particular systems with
partially measurable states. We discuss the importance of modeling both the
measurement system and the state estimators during policy optimization. The
effectiveness of the proposed solutions has been tested in simulation and in
two real systems, a Furuta pendulum and a ball-and-plate.
