Towards Understanding the Transferability of Deep Representations
  Deep neural networks trained on a wide range of datasets demonstrate
impressive transferability. Deep features appear general in that they are
applicable to many datasets and tasks. Such property is in prevalent use in
real-world applications. A neural network pretrained on large datasets, such as
ImageNet, can significantly boost generalization and accelerate training if
fine-tuned to a smaller target dataset. Despite its pervasiveness, few effort
has been devoted to uncovering the reason of transferability in deep feature
representations. This paper tries to understand transferability from the
perspectives of improved generalization, optimization and the feasibility of
transferability. We demonstrate that 1) Transferred models tend to find flatter
minima, since their weight matrices stay close to the original flat region of
pretrained parameters when transferred to a similar target dataset; 2)
Transferred representations make the loss landscape more favorable with
improved Lipschitzness, which accelerates and stabilizes training
substantially. The improvement largely attributes to the fact that the
principal component of gradient is suppressed in the pretrained parameters,
thus stabilizing the magnitude of gradient in back-propagation. 3) The
feasibility of transferability is related to the similarity of both input and
label. And a surprising discovery is that the feasibility is also impacted by
the training stages in that the transferability first increases during
training, and then declines. We further provide a theoretical analysis to
verify our observations.
