Smooth Loss Functions for Deep Top-k Classification
  The top-k error is a common measure of performance in machine learning and
computer vision. In practice, top-k classification is typically performed with
deep neural networks trained with the cross-entropy loss. Theoretical results
indeed suggest that cross-entropy is an optimal learning objective for such a
task in the limit of infinite data. In the context of limited and noisy data
however, the use of a loss function that is specifically designed for top-k
classification can bring significant improvements. Our empirical evidence
suggests that the loss function must be smooth and have non-sparse gradients in
order to work well with deep neural networks. Consequently, we introduce a
family of smoothed loss functions that are suited to top-k optimization via
deep learning. The widely used cross-entropy is a special case of our family.
Evaluating our smooth loss functions is computationally challenging: a na\"ive
algorithm would require $\mathcal{O}(\binom{n}{k})$ operations, where n is the
number of classes. Thanks to a connection to polynomial algebra and a
divide-and-conquer approach, we provide an algorithm with a time complexity of
$\mathcal{O}(k n)$. Furthermore, we present a novel approximation to obtain
fast and stable algorithms on GPUs with single floating point precision. We
compare the performance of the cross-entropy loss and our margin-based losses
in various regimes of noise and data size, for the predominant use case of k=5.
Our investigation reveals that our loss is more robust to noise and overfitting
than cross-entropy.
