Generalisation error in learning with random features and the hidden
  manifold model
  We study generalised linear regression and classification for a synthetically
generated dataset encompassing different problems of interest, such as learning
with random features, neural networks in the lazy training regime, and the
hidden manifold model. We consider the high-dimensional regime and using the
replica method from statistical physics, we provide a closed-form expression
for the asymptotic generalisation performance in these problems, valid in both
the under- and over-parametrised regimes and for a broad choice of generalised
linear model loss functions. In particular, we show how to obtain analytically
the so-called double descent behaviour for logistic regression with a peak at
the interpolation threshold, we illustrate the superiority of orthogonal
against random Gaussian projections in learning with random features, and
discuss the role played by correlations in the data generated by the hidden
manifold model. Beyond the interest in these particular problems, the
theoretical formalism introduced in this manuscript provides a path to further
extensions to more complex tasks.
