A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth
  Trade-Off
  Reducing the precision of weights and activation functions in neural network
training, with minimal impact on performance, is essential for the deployment
of these models in resource-constrained environments. We apply mean-field
techniques to networks with quantized activations in order to evaluate the
degree to which quantization degrades signal propagation at initialization. We
derive initialization schemes which maximize signal propagation in such
networks and suggest why this is helpful for generalization. Building on these
results, we obtain a closed form implicit equation for $L_{\max}$, the maximal
trainable depth (and hence model capacity), given $N$, the number of
quantization levels in the activation function. Solving this equation
numerically, we obtain asymptotically: $L_{\max}\propto N^{1.82}$.
