Deep learning as optimal control problems: models and numerical methods
  We consider recent work of Haber and Ruthotto 2017 and Chang et al. 2018,
where deep learning neural networks have been interpreted as discretisations of
an optimal control problem subject to an ordinary differential equation
constraint. We review the first order conditions for optimality, and the
conditions ensuring optimality after discretisation. This leads to a class of
algorithms for solving the discrete optimal control problem which guarantee
that the corresponding discrete necessary conditions for optimality are
fulfilled. The differential equation setting lends itself to learning
additional parameters such as the time discretisation. We explore this
extension alongside natural constraints (e.g. time steps lie in a simplex). We
compare these deep learning algorithms numerically in terms of induced flow and
generalisation ability.
