Neuron Campaign for Initialization Guided by Information Bottleneck
  Theory
  Initialization plays a critical role in the training of deep neural networks
(DNN). Existing initialization strategies mainly focus on stabilizing the
training process to mitigate gradient vanish/explosion problems. However, these
initialization methods are lacking in consideration about how to enhance
generalization ability. The Information Bottleneck (IB) theory is a well-known
understanding framework to provide an explanation about the generalization of
DNN. Guided by the insights provided by IB theory, we design two criteria for
better initializing DNN. And we further design a neuron campaign initialization
algorithm to efficiently select a good initialization for a neural network on a
given dataset. The experiments on MNIST dataset show that our method can lead
to a better generalization performance with faster convergence.
