How does topology influence gradient propagation and model performance
  of deep networks with DenseNet-type skip connections?
  DenseNets introduce concatenation-type skip connections that achieve
state-of-the-art accuracy in several computer vision tasks. In this paper, we
reveal that the topology of the concatenation-type skip connections is closely
related to the gradient propagation which, in turn, enables a predictable
behavior of DNNs' test performance. To this end, we introduce a new metric
called NN-Mass to quantify how effectively information flows through DNNs.
Moreover, we empirically show that NN-Mass also works for other types of skip
connections, e.g., for ResNets, Wide-ResNets (WRNs), and MobileNets, which
contain addition-type skip connections (i.e., residuals or inverted residuals).
As such, for both DenseNet-like CNNs and ResNets/WRNs/MobileNets, our
theoretically grounded NN-Mass can identify models with similar accuracy,
despite having significantly different size/compute requirements. Detailed
experiments on both synthetic and real datasets (e.g., MNIST, CIFAR-10,
CIFAR-100, ImageNet) provide extensive evidence for our insights. Finally, the
closed-form equation of our NN-Mass enables us to design significantly
compressed DenseNets (for CIFAR-10) and MobileNets (for ImageNet) directly at
initialization without time-consuming training and/or searching.
