Dynamic Neural Diversification: Path to Computationally Sustainable
  Neural Networks
  Small neural networks with a constrained number of trainable parameters, can
be suitable resource-efficient candidates for many simple tasks, where now
excessively large models are used. However, such models face several problems
during the learning process, mainly due to the redundancy of the individual
neurons, which results in sub-optimal accuracy or the need for additional
training steps. Here, we explore the diversity of the neurons within the hidden
layer during the learning process, and analyze how the diversity of the neurons
affects predictions of the model. As following, we introduce several techniques
to dynamically reinforce diversity between neurons during the training. These
decorrelation techniques improve learning at early stages and occasionally help
to overcome local minima faster. Additionally, we describe novel weight
initialization method to obtain decorrelated, yet stochastic weight
initialization for a fast and efficient neural network training. Decorrelated
weight initialization in our case shows about 40% relative increase in test
accuracy during the first 5 epochs.
