Frivolous Units: Wider Networks Are Not Really That Wide
  A remarkable characteristic of overparameterized deep neural networks (DNNs)
is that their accuracy does not degrade when the network's width is increased.
Recent evidence suggests that developing compressible representations is key
for adjusting the complexity of large networks to the learning task at hand.
However, these compressible representations are poorly understood. A promising
strand of research inspired from biology is understanding representations at
the unit level as it offers a more granular and intuitive interpretation of the
neural mechanisms. In order to better understand what facilitates increases in
width without decreases in accuracy, we ask: Are there mechanisms at the unit
level by which networks control their effective complexity as their width is
increased? If so, how do these depend on the architecture, dataset, and
training parameters? We identify two distinct types of "frivolous" units that
proliferate when the network's width is increased: prunable units which can be
dropped out of the network without significant change to the output and
redundant units whose activities can be expressed as a linear combination of
others. These units imply complexity constraints as the function the network
represents could be expressed by a network without them. We also identify how
the development of these units can be influenced by architecture and a number
of training factors. Together, these results help to explain why the accuracy
of DNNs does not degrade when width is increased and highlight the importance
of frivolous units toward understanding implicit regularization in DNNs.
