Activation function dependence of the storage capacity of treelike
  neural networks
  The expressive power of artificial neural networks crucially depends on the
nonlinearity of their activation functions. Though a wide variety of nonlinear
activation functions have been proposed for use in artificial neural networks,
a detailed understanding of their role in determining the expressive power of a
network has not emerged. Here, we study how activation functions affect the
storage capacity of treelike two-layer networks. We relate the boundedness or
divergence of the capacity in the infinite-width limit to the smoothness of the
activation function, elucidating the relationship between previously studied
special cases. Our results show that nonlinearity can both increase capacity
and decrease the robustness of classification, and provide simple estimates for
the capacity of networks with several commonly used activation functions.
Furthermore, they generate a hypothesis for the functional benefit of dendritic
spikes in branched neurons.
