Finding the Needle in the Haystack with Convolutions: on the benefits of
  architectural bias
  Despite the phenomenal success of deep neural networks in a broad range of
learning tasks, there is a lack of theory to understand the way they work. In
particular, Convolutional Neural Networks (CNNs) are known to perform much
better than Fully-Connected Networks (FCNs) on spatially structured data: the
architectural structure of CNNs benefits from prior knowledge on the features
of the data, for instance their translation invariance. The aim of this work is
to understand this fact through the lens of dynamics in the loss landscape.
  We introduce a method that maps a CNN to its equivalent FCN (denoted as
eFCN). Such an embedding enables the comparison of CNN and FCN training
dynamics directly in the FCN space. We use this method to test a new training
protocol, which consists in training a CNN, embedding it to FCN space at a
certain ``relax time'', then resuming the training in FCN space. We observe
that for all relax times, the deviation from the CNN subspace is small, and the
final performance reached by the eFCN is higher than that reachable by a
standard FCN of same architecture. More surprisingly, for some intermediate
relax times, the eFCN outperforms the CNN it stemmed, by combining the prior
information of the CNN and the expressivity of the FCN in a complementary way.
The practical interest of our protocol is limited by the very large size of the
highly sparse eFCN. However, it offers interesting insights into the
persistence of architectural bias under stochastic gradient dynamics. It shows
the existence of some rare basins in the FCN loss landscape associated with
very good generalization. These can only be accessed thanks to the CNN prior,
which helps navigate the landscape during the early stages of optimization.
