Improved Exploration through Latent Trajectory Optimization in Deep
  Deterministic Policy Gradient
  Model-free reinforcement learning algorithms such as Deep Deterministic
Policy Gradient (DDPG) often require additional exploration strategies,
especially if the actor is of deterministic nature. This work evaluates the use
of model-based trajectory optimization methods used for exploration in Deep
Deterministic Policy Gradient when trained on a latent image embedding. In
addition, an extension of DDPG is derived using a value function as critic,
making use of a learned deep dynamics model to compute the policy gradient.
This approach leads to a symbiotic relationship between the deep reinforcement
learning algorithm and the latent trajectory optimizer. The trajectory
optimizer benefits from the critic learned by the RL algorithm and the latter
from the enhanced exploration generated by the planner. The developed methods
are evaluated on two continuous control tasks, one in simulation and one in the
real world. In particular, a Baxter robot is trained to perform an insertion
task, while only receiving sparse rewards and images as observations from the
environment.
