What Do Neural Networks Learn When Trained With Random Labels?
  We study deep neural networks (DNNs) trained on natural image data with
entirely random labels. Despite its popularity in the literature, where it is
often used to study memorization, generalization, and other phenomena, little
is known about what DNNs learn in this setting. In this paper, we show
analytically for convolutional and fully connected networks that an alignment
between the principal components of network parameters and data takes place
when training with random labels. We study this alignment effect by
investigating neural networks pre-trained on randomly labelled image data and
subsequently fine-tuned on disjoint datasets with random or real labels. We
show how this alignment produces a positive transfer: networks pre-trained with
random labels train faster downstream compared to training from scratch even
after accounting for simple effects, such as weight scaling. We analyze how
competing effects, such as specialization at later layers, may hide the
positive transfer. These effects are studied in several network architectures,
including VGG16 and ResNet18, on CIFAR10 and ImageNet.
