High Dimensional Classification via Regularized and Unregularized
  Empirical Risk Minimization: Precise Error and Optimal Loss
  This article provides, through theoretical analysis, an in-depth
understanding of the classification performance of the empirical risk
minimization framework, in both ridge-regularized and unregularized cases, when
high dimensional data are considered. Focusing on the fundamental problem of
separating a two-class Gaussian mixture, the proposed analysis allows for a
precise prediction of the classification error for a set of numerous data
vectors $\mathbf{x} \in \mathbb R^p$ of sufficiently large dimension $p$. This
precise error depends on the loss function, the number of training samples, and
the statistics of the mixture data model. It is shown to hold beyond Gaussian
distribution under some additional non-sparsity condition of the data
statistics. Building upon this quantitative error analysis, we identify the
simple square loss as the optimal choice for high dimensional classification in
both ridge-regularized and unregularized cases, regardless of the number of
training samples.
