On the interplay between data structure and loss function in
  classification problems
  One of the central puzzles in modern machine learning is the ability of
heavily overparametrized models to generalize well. Although the
low-dimensional structure of typical datasets is key to this behavior, most
theoretical studies of overparametrization focus on isotropic inputs. In this
work, we instead consider an analytically tractable model of structured data,
where the input covariance is built from independent blocks allowing us to tune
the saliency of low-dimensional structures and their alignment with respect to
the target function. Using methods from statistical physics, we derive a
precise asymptotic expression for the train and test error achieved by random
feature models trained to classify such data, which is valid for any convex
loss function. We study in detail how the data structure affects the double
descent curve, and show that in the over-parametrized regime, its impact is
greater for logistic loss than for mean-squared loss: the easier the task, the
wider the gap in performance at the advantage of the logistic loss. Our
insights are confirmed by numerical experiments on MNIST and CIFAR10.
