Convergence of Deep ReLU Networks
  We explore convergence of deep neural networks with the popular ReLU
activation function, as the depth of the networks tends to infinity. To this
end, we introduce the notion of activation domains and activation matrices of a
ReLU network. By replacing applications of the ReLU activation function by
multiplications with activation matrices on activation domains, we obtain an
explicit expression of the ReLU network. We then identify the convergence of
the ReLU networks as convergence of a class of infinite products of matrices.
Sufficient and necessary conditions for convergence of these infinite products
of matrices are studied. As a result, we establish necessary conditions for
ReLU networks to converge that the sequence of weight matrices converges to the
identity matrix and the sequence of the bias vectors converges to zero as the
depth of ReLU networks increases to infinity. Moreover, we obtain sufficient
conditions in terms of the weight matrices and bias vectors at hidden layers
for pointwise convergence of deep ReLU networks. These results provide
mathematical insights to the design strategy of the well-known deep residual
networks in image classification.
