Topology Reduction in Deep Convolutional Feature Extraction Networks
  Deep convolutional neural networks (CNNs) used in practice employ potentially
hundreds of layers and $10$,$000$s of nodes. Such network sizes entail
significant computational complexity due to the large number of convolutions
that need to be carried out; in addition, a large number of parameters needs to
be learned and stored. Very deep and wide CNNs may therefore not be well suited
to applications operating under severe resource constraints as is the case,
e.g., in low-power embedded and mobile platforms. This paper aims at
understanding the impact of CNN topology, specifically depth and width, on the
network's feature extraction capabilities. We address this question for the
class of scattering networks that employ either Weyl-Heisenberg filters or
wavelets, the modulus non-linearity, and no pooling. The exponential feature
map energy decay results in Wiatowski et al., 2017, are generalized to
$\mathcal{O}(a^{-N})$, where an arbitrary decay factor $a>1$ can be realized
through suitable choice of the Weyl-Heisenberg prototype function or the mother
wavelet. We then show how networks of fixed (possibly small) depth $N$ can be
designed to guarantee that $((1-\varepsilon)\cdot 100)\%$ of the input signal's
energy are contained in the feature vector. Based on the notion of
operationally significant nodes, we characterize, partly rigorously and partly
heuristically, the topology-reducing effects of (effectively) band-limited
input signals, band-limited filters, and feature map symmetries. Finally, for
networks based on Weyl-Heisenberg filters, we determine the prototype function
bandwidth that minimizes---for fixed network depth $N$---the average number of
operationally significant nodes per layer.
