Why Do Better Loss Functions Lead to Less Transferable Features?
  Previous work has proposed many new loss functions and regularizers that
improve test accuracy on image classification tasks. However, it is not clear
whether these loss functions learn better representations for downstream tasks.
This paper studies how the choice of training objective affects the
transferability of the hidden representations of convolutional neural networks
trained on ImageNet. We show that many objectives lead to statistically
significant improvements in ImageNet accuracy over vanilla softmax
cross-entropy, but the resulting fixed feature extractors transfer
substantially worse to downstream tasks, and the choice of loss has little
effect when networks are fully fine-tuned on the new tasks. Using centered
kernel alignment to measure similarity between hidden representations of
networks, we find that differences among loss functions are apparent only in
the last few layers of the network. We delve deeper into representations of the
penultimate layer, finding that different objectives and hyperparameter
combinations lead to dramatically different levels of class separation.
Representations with higher class separation obtain higher accuracy on the
original task, but their features are less useful for downstream tasks. Our
results suggest there exists a trade-off between learning invariant features
for the original task and features relevant for transfer tasks.
