Convolutional Normalization
  As the deep neural networks are being applied to complex tasks, the size of
the networks and architecture increases and their topology becomes more
complicated too. At the same time, training becomes slow and at some instances
inefficient. This motivated the introduction of various normalization
techniques such as Batch Normalization and Layer Normalization. The
aforementioned normalization methods use arithmetic operations to compute an
approximation statistics (mainly the first and second moments) of the layer's
data and use it to normalize it. The aforementioned methods use plain Monte
Carlo method to approximate the statistics and such method fails when
approximating the statistics whose distribution is complex. Here, we propose an
approach that uses weighted sum, implemented using depth-wise convolutional
neural networks, to not only approximate the statistics, but to learn the
coefficients of the sum.
