A proof of convergence for gradient descent in the training of
  artificial neural networks for constant target functions
  Gradient descent optimization algorithms are the standard ingredients that
are used to train artificial neural networks (ANNs). Even though a huge number
of numerical simulations indicate that gradient descent optimization methods do
indeed convergence in the training of ANNs, until today there is no rigorous
theoretical analysis which proves (or disproves) this conjecture. In
particular, even in the case of the most basic variant of gradient descent
optimization algorithms, the plain vanilla gradient descent method, it remains
an open problem to prove or disprove the conjecture that gradient descent
converges in the training of ANNs. In this article we solve this problem in the
special situation where the target function under consideration is a constant
function. More specifically, in the case of constant target functions we prove
in the training of rectified fully-connected feedforward ANNs with one-hidden
layer that the risk function of the gradient descent method does indeed
converge to zero. Our mathematical analysis strongly exploits the property that
the rectifier function is the activation function used in the considered ANNs.
A key contribution of this work is to explicitly specify a Lyapunov function
for the gradient flow system of the ANN parameters. This Lyapunov function is
the central tool in our convergence proof of the gradient descent method.
