A Mathematical Theory of Deep Convolutional Neural Networks for Feature
  Extraction
  Deep convolutional neural networks have led to breakthrough results in
numerous practical machine learning tasks such as classification of images in
the ImageNet data set, control-policy-learning to play Atari games or the board
game Go, and image captioning. Many of these applications first perform feature
extraction and then feed the results thereof into a trainable classifier. The
mathematical analysis of deep convolutional neural networks for feature
extraction was initiated by Mallat, 2012. Specifically, Mallat considered
so-called scattering networks based on a wavelet transform followed by the
modulus non-linearity in each network layer, and proved translation invariance
(asymptotically in the wavelet scale parameter) and deformation stability of
the corresponding feature extractor. This paper complements Mallat's results by
developing a theory that encompasses general convolutional transforms, or in
more technical parlance, general semi-discrete frames (including
Weyl-Heisenberg filters, curvelets, shearlets, ridgelets, wavelets, and learned
filters), general Lipschitz-continuous non-linearities (e.g., rectified linear
units, shifted logistic sigmoids, hyperbolic tangents, and modulus functions),
and general Lipschitz-continuous pooling operators emulating, e.g.,
sub-sampling and averaging. In addition, all of these elements can be different
in different network layers. For the resulting feature extractor we prove a
translation invariance result of vertical nature in the sense of the features
becoming progressively more translation-invariant with increasing network
depth, and we establish deformation sensitivity bounds that apply to signal
classes such as, e.g., band-limited functions, cartoon functions, and Lipschitz
functions.
