A Model of Double Descent for High-dimensional Binary Linear
  Classification
  We consider a model for logistic regression where only a subset of features
of size $p$ is used for training a linear classifier over $n$ training samples.
The classifier is obtained by running gradient descent (GD) on logistic loss.
For this model, we investigate the dependence of the classification error on
the overparameterization ratio $\kappa=p/n$. First, building on known
deterministic results on the implicit bias of GD, we uncover a phase-transition
phenomenon for the case of Gaussian features: the classification error of GD is
the same as that of the maximum-likelihood (ML) solution when
$\kappa<\kappa_\star$, and that of the max-margin (SVM) solution when
$\kappa>\kappa_\star$. Next, using the convex Gaussian min-max theorem (CGMT),
we sharply characterize the performance of both the ML and the SVM solutions.
Combining these results, we obtain curves that explicitly characterize the
classification error for varying values of $\kappa$. The numerical results
validate the theoretical predictions and unveil double-descent phenomena that
complement similar recent findings in linear regression settings as well as
empirical observations in more complex learning scenarios.
