Where Should We Begin? A Low-Level Exploration of Weight Initialization
  Impact on Quantized Behaviour of Deep Neural Networks
  With the proliferation of deep convolutional neural network (CNN) algorithms
for mobile processing, limited precision quantization has become an essential
tool for CNN efficiency. Consequently, various works have sought to design
fixed precision quantization algorithms and quantization-focused optimization
techniques that minimize quantization induced performance degradation. However,
there is little concrete understanding of how various CNN design decisions/best
practices affect quantized inference behaviour. Weight initialization
strategies are often associated with solving issues such as vanishing/exploding
gradients but an often-overlooked aspect is their impact on the final trained
distributions of each layer. We present an in-depth, fine-grained ablation
study of the effect of different weights initializations on the final
distributions of weights and activations of different CNN architectures. The
fine-grained, layerwise analysis enables us to gain deep insights on how
initial weights distributions will affect final accuracy and quantized
behaviour. To our best knowledge, we are the first to perform such a low-level,
in-depth quantitative analysis of weights initialization and its effect on
quantized behaviour.
