Learning the optimal state-feedback via supervised imitation learning
  Imitation learning is a control design paradigm that seeks to learn a control
policy reproducing demonstrations from expert agents. By substituting expert
demonstrations for optimal behaviours, the same paradigm leads to the design of
control policies closely approximating the optimal state-feedback. This
approach requires training a machine learning algorithm (in our case deep
neural networks) directly on state-control pairs originating from optimal
trajectories. We have shown in previous work that, when restricted to
low-dimensional state and control spaces, this approach is very successful in
several deterministic, non-linear problems in continuous-time. In this work, we
refine our previous studies using as a test case a simple quadcopter model with
quadratic and time-optimal objective functions. We describe in detail the best
learning pipeline we have developed, that is able to approximate via deep
neural networks the state-feedback map to a very high accuracy. We introduce
the use of the softplus activation function in the hidden units of neural
networks showing that it results in a smoother control profile whilst retaining
the benefits of rectifiers. We show how to evaluate the optimality of the
trained state-feedback, and find that already with two layers the objective
function reached and its optimal value differ by less than one percent. We
later consider also an additional metric linked to the system asymptotic
behaviour - time taken to converge to the policy's fixed point. With respect to
these metrics, we show that improvements in the mean absolute error do not
necessarily correspond to better policies.
