Variability of Artificial Neural Networks
  What makes an artificial neural network easier to train and more likely to
produce desirable solutions than other comparable networks? In this paper, we
provide a new angle to study such issues under the setting of a fixed number of
model parameters which in general is the most dominant cost factor. We
introduce a notion of variability and show that it correlates positively to the
activation ratio and negatively to a phenomenon called {Collapse to Constants}
(or C2C), which is closely related but not identical to the phenomenon commonly
known as vanishing gradient. Experiments on a styled model problem empirically
verify that variability is indeed a key performance indicator for fully
connected neural networks. The insights gained from this variability study will
help the design of new and effective neural network architectures.
