Adaptive Signal Variances: CNN Initialization Through Modern
  Architectures
  Deep convolutional neural networks (CNN) have achieved the unwavering
confidence in its performance on image processing tasks. The CNN architecture
constitutes a variety of different types of layers including the convolution
layer and the max-pooling layer. CNN practitioners widely understand the fact
that the stability of learning depends on how to initialize the model
parameters in each layer. Nowadays, no one doubts that the de facto standard
scheme for initialization is the so-called Kaiming initialization that has been
developed by He et al. The Kaiming scheme was derived from a much simpler model
than the currently used CNN structure having evolved since the emergence of the
Kaiming scheme. The Kaiming model consists only of the convolution and fully
connected layers, ignoring the max-pooling layer and the global average pooling
layer. In this study, we derived the initialization scheme again not from the
simplified Kaiming model, but precisely from the modern CNN architectures, and
empirically investigated how the new initialization method performs compared to
the de facto standard ones that are widely used today.
