Learning distinct features helps, provably
  We study the diversity of the features learned by a two-layer neural network
trained with the least squares loss. We measure the diversity by the average
$L_2$-distance between the hidden-layer features and theoretically investigate
how learning non-redundant distinct features affects the performance of the
network. To do so, we derive novel generalization bounds depending on feature
diversity based on Rademacher complexity for such networks. Our analysis proves
that more distinct features at the network's hidden units within the
intermediate layer lead to better generalization. We also show how to extend
our results to deeper networks and different losses.
