Cooperative Initialization based Deep Neural Network Training
  Researchers have proposed various activation functions. These activation
functions help the deep network to learn non-linear behavior with a significant
effect on training dynamics and task performance. The performance of these
activations also depends on the initial state of the weight parameters, i.e.,
different initial state leads to a difference in the performance of a network.
In this paper, we have proposed a cooperative initialization for training the
deep network using ReLU activation function to improve the network performance.
Our approach uses multiple activation functions in the initial few epochs for
the update of all sets of weight parameters while training the network. These
activation functions cooperate to overcome their drawbacks in the update of
weight parameters, which in effect learn better "feature representation" and
boost the network performance later. Cooperative initialization based training
also helps in reducing the overfitting problem and does not increase the number
of parameters, inference (test) time in the final model while improving the
performance. Experiments show that our approach outperforms various baselines
and, at the same time, performs well over various tasks such as classification
and detection. The Top-1 classification accuracy of the model trained using our
approach improves by 2.8% for VGG-16 and 2.1% for ResNet-56 on CIFAR-100
dataset.
