Convergence Analysis of Deep Residual Networks
  Various powerful deep neural network architectures have made great
contribution to the exciting successes of deep learning in the past two
decades. Among them, deep Residual Networks (ResNets) are of particular
importance because they demonstrated great usefulness in computer vision by
winning the first place in many deep learning competitions. Also, ResNets were
the first class of neural networks in the development history of deep learning
that are really deep. It is of mathematical interest and practical meaning to
understand the convergence of deep ResNets. We aim at characterizing the
convergence of deep ResNets as the depth tends to infinity in terms of the
parameters of the networks. Toward this purpose, we first give a matrix-vector
description of general deep neural networks with shortcut connections and
formulate an explicit expression for the networks by using the notions of
activation domains and activation matrices. The convergence is then reduced to
the convergence of two series involving infinite products of non-square
matrices. By studying the two series, we establish a sufficient condition for
pointwise convergence of ResNets. Our result is able to give justification for
the design of ResNets. We also conduct experiments on benchmark machine
learning data to verify our results.
