Deep Learning on Small Datasets without Pre-Training using Cosine Loss
  Two things seem to be indisputable in the contemporary deep learning
discourse: 1. The categorical cross-entropy loss after softmax activation is
the method of choice for classification. 2. Training a CNN classifier from
scratch on small datasets does not work well. In contrast to this, we show that
the cosine loss function provides significantly better performance than
cross-entropy on datasets with only a handful of samples per class. For
example, the accuracy achieved on the CUB-200-2011 dataset without pre-training
is by 30% higher than with the cross-entropy loss. Further experiments on other
popular datasets confirm our findings. Moreover, we demonstrate that
integrating prior knowledge in the form of class hierarchies is straightforward
with the cosine loss and improves classification performance further.
