Towards Understanding Residual and Dilated Dense Neural Networks via
  Convolutional Sparse Coding
  Convolutional neural network (CNN) and its variants have led to many
state-of-art results in various fields. However, a clear theoretical
understanding about them is still lacking. Recently, multi-layer convolutional
sparse coding (ML-CSC) has been proposed and proved to equal such simply
stacked networks (plain networks). Here, we think three factors in each layer
of it including the initialization, the dictionary design and the number of
iterations greatly affect the performance of ML-CSC. Inspired by these
considerations, we propose two novel multi-layer models--residual convolutional
sparse coding model (Res-CSC) and mixed-scale dense convolutional sparse coding
model (MSD-CSC), which have close relationship with the residual neural network
(ResNet) and mixed-scale (dilated) dense neural network (MSDNet), respectively.
Mathematically, we derive the shortcut connection in ResNet as a special case
of a new forward propagation rule on ML-CSC. We find a theoretical
interpretation of the dilated convolution and dense connection in MSDNet by
analyzing MSD-CSC, which gives a clear mathematical understanding about them.
We implement the iterative soft thresholding algorithm (ISTA) and its fast
version to solve Res-CSC and MSD-CSC, which can employ the unfolding operation
for further improvements. At last, extensive numerical experiments and
comparison with competing methods demonstrate their effectiveness using three
typical datasets.
