Sharp Rate of Convergence for Deep Neural Network Classifiers under the
  Teacher-Student Setting
  Classifiers built with neural networks handle large-scale high dimensional
data, such as facial images from computer vision, extremely well while
traditional statistical methods often fail miserably. In this paper, we attempt
to understand this empirical success in high dimensional classification by
deriving the convergence rates of excess risk. In particular, a teacher-student
framework is proposed that assumes the Bayes classifier to be expressed as ReLU
neural networks. In this setup, we obtain a sharp rate of convergence, i.e.,
$\tilde{O}_d(n^{-2/3})$, for classifiers trained using either 0-1 loss or hinge
loss. This rate can be further improved to $\tilde{O}_d(n^{-1})$ when the data
distribution is separable. Here, $n$ denotes the sample size. An interesting
observation is that the data dimension only contributes to the $\log(n)$ term
in the above rates. This may provide one theoretical explanation for the
empirical successes of deep neural networks in high dimensional classification,
particularly for structured data.
