Asymptotic Singular Value Distribution of Linear Convolutional Layers
  In convolutional neural networks, the linear transformation of multi-channel
two-dimensional convolutional layers with linear convolution is a block matrix
with doubly Toeplitz blocks. Although a "wrapping around" operation can
transform linear convolution to a circular one, by which the singular values
can be approximated with reduced computational complexity by those of a block
matrix with doubly circulant blocks, the accuracy of such an approximation is
not guaranteed. In this paper, we propose to inspect such a linear
transformation matrix through its asymptotic spectral representation - the
spectral density matrix - by which we develop a simple singular value
approximation method with improved accuracy over the circular approximation, as
well as upper bounds for spectral norm with reduced computational complexity.
Compared with the circular approximation, we obtain moderate improvement with a
subtle adjustment of the singular value distribution. We also demonstrate that
the spectral norm upper bounds are effective spectral regularizers for
improving generalization performance in ResNets.
