VC dimension of partially quantized neural networks in the
  overparametrized regime
  Vapnik-Chervonenkis (VC) theory has so far been unable to explain the small
generalization error of overparametrized neural networks. Indeed, existing
applications of VC theory to large networks obtain upper bounds on VC dimension
that are proportional to the number of weights, and for a large class of
networks, these upper bound are known to be tight. In this work, we focus on a
class of partially quantized networks that we refer to as hyperplane
arrangement neural networks (HANNs). Using a sample compression analysis, we
show that HANNs can have VC dimension significantly smaller than the number of
weights, while being highly expressive. In particular, empirical risk
minimization over HANNs in the overparametrized regime achieves the minimax
rate for classification with Lipschitz posterior class probability. We further
demonstrate the expressivity of HANNs empirically. On a panel of 121 UCI
datasets, overparametrized HANNs match the performance of state-of-the-art
full-precision models.
