Beyond Folklore: A Scaling Calculus for the Design and Initialization of
  ReLU Networks
  We propose a system for calculating a "scaling constant" for layers and
weights of neural networks. We relate this scaling constant to two important
quantities that relate to the optimizability of neural networks, and argue that
a network that is "preconditioned" via scaling, in the sense that all weights
have the same scaling constant, will be easier to train. This scaling calculus
results in a number of consequences, among them the fact that the geometric
mean of the fan-in and fan-out, rather than the fan-in, fan-out, or arithmetic
mean, should be used for the initialization of the variance of weights in a
neural network. Our system allows for the off-line design & engineering of ReLU
neural networks, potentially replacing blind experimentation.
