Learning curves for the multi-class teacher-student perceptron
  One of the most classical results in high-dimensional learning theory
provides a closed-form expression for the generalisation error of binary
classification with the single-layer teacher-student perceptron on i.i.d.
Gaussian inputs. Both Bayes-optimal estimation and empirical risk minimisation
(ERM) were extensively analysed for this setting. At the same time, a
considerable part of modern machine learning practice concerns multi-class
classification. Yet, an analogous analysis for the corresponding multi-class
teacher-student perceptron was missing. In this manuscript we fill this gap by
deriving and evaluating asymptotic expressions for both the Bayes-optimal and
ERM generalisation errors in the high-dimensional regime. For Gaussian teacher
weights, we investigate the performance of ERM with both cross-entropy and
square losses, and explore the role of ridge regularisation in approaching
Bayes-optimality. In particular, we observe that regularised cross-entropy
minimisation yields close-to-optimal accuracy. Instead, for a binary teacher we
show that a first-order phase transition arises in the Bayes-optimal
performance.
