Optimal Errors and Phase Transitions in High-Dimensional Generalized
  Linear Models
  Generalized linear models (GLMs) arise in high-dimensional machine learning,
statistics, communications and signal processing. In this paper we analyze GLMs
when the data matrix is random, as relevant in problems such as compressed
sensing, error-correcting codes or benchmark models in neural networks. We
evaluate the mutual information (or "free entropy") from which we deduce the
Bayes-optimal estimation and generalization errors. Our analysis applies to the
high-dimensional limit where both the number of samples and the dimension are
large and their ratio is fixed. Non-rigorous predictions for the optimal errors
existed for special cases of GLMs, e.g. for the perceptron, in the field of
statistical physics based on the so-called replica method. Our present paper
rigorously establishes those decades old conjectures and brings forward their
algorithmic interpretation in terms of performance of the generalized
approximate message-passing algorithm. Furthermore, we tightly characterize,
for many learning problems, regions of parameters for which this algorithm
achieves the optimal performance, and locate the associated sharp phase
transitions separating learnable and non-learnable regions. We believe that
this random version of GLMs can serve as a challenging benchmark for
multi-purpose algorithms. This paper is divided in two parts that can be read
independently: The first part (main part) presents the model and main results,
discusses some applications and sketches the main ideas of the proof. The
second part (supplementary informations) is much more detailed and provides
more examples as well as all the proofs.
