Direct Random Search for Fine Tuning of Deep Reinforcement Learning
  Policies
  Researchers have demonstrated that Deep Reinforcement Learning (DRL) is a
powerful tool for finding policies that perform well on complex robotic
systems. However, these policies are often unpredictable and can induce highly
variable behavior when evaluated with only slightly different initial
conditions. Training considerations constrain DRL algorithm designs in that
most algorithms must use stochastic policies during training. The resulting
policy used during deployment, however, can and frequently is a deterministic
one that uses the Maximum Likelihood Action (MLA) at each step. In this work,
we show that a direct random search is very effective at fine-tuning DRL
policies by directly optimizing them using deterministic rollouts. We
illustrate this across a large collection of reinforcement learning
environments, using a wide variety of policies obtained from different
algorithms. Our results show that this method yields more consistent and higher
performing agents on the environments we tested. Furthermore, we demonstrate
how this method can be used to extend our previous work on shrinking the
dimensionality of the reachable state space of closed-loop systems run under
Deep Neural Network (DNN) policies.
