Phase Collapse in Neural Networks
  Deep convolutional classifiers linearly separate image classes and improve
accuracy as depth increases. They progressively reduce the spatial dimension
whereas the number of channels grows with depth. Spatial variability is
therefore transformed into variability along channels. A fundamental challenge
is to understand the role of non-linearities together with convolutional
filters in this transformation. ReLUs with biases are often interpreted as
thresholding operators that improve discrimination through sparsity. This paper
demonstrates that it is a different mechanism called phase collapse which
eliminates spatial variability while linearly separating classes. We show that
collapsing the phases of complex wavelet coefficients is sufficient to reach
the classification accuracy of ResNets of similar depths. However, replacing
the phase collapses with thresholding operators that enforce sparsity
considerably degrades the performance. We explain these numerical results by
showing that the iteration of phase collapses progressively improves separation
of classes, as opposed to thresholding non-linearities.
