Reinforcement Learning for Robust Missile Autopilot Design
  Designing missiles' autopilot controllers has been a complex task, given the
extensive flight envelope and the nonlinear flight dynamics. A solution that
can excel both in nominal performance and in robustness to uncertainties is
still to be found. While Control Theory often debouches into parameters'
scheduling procedures, Reinforcement Learning has presented interesting results
in ever more complex tasks, going from videogames to robotic tasks with
continuous action domains. However, it still lacks clearer insights on how to
find adequate reward functions and exploration strategies. To the best of our
knowledge, this work is pioneer in proposing Reinforcement Learning as a
framework for flight control. In fact, it aims at training a model-free agent
that can control the longitudinal flight of a missile, achieving optimal
performance and robustness to uncertainties. To that end, under TRPO's
methodology, the collected experience is augmented according to HER, stored in
a replay buffer and sampled according to its significance. Not only does this
work enhance the concept of prioritized experience replay into BPER, but it
also reformulates HER, activating them both only when the training progress
converges to suboptimal policies, in what is proposed as the SER methodology.
Besides, the Reward Engineering process is carefully detailed. The results show
that it is possible both to achieve the optimal performance and to improve the
agent's robustness to uncertainties (with low damage on nominal performance) by
further training it in non-nominal environments, therefore validating the
proposed approach and encouraging future research in this field.
