GradMax: Growing Neural Networks using Gradient Information
  The architecture and the parameters of neural networks are often optimized
independently, which requires costly retraining of the parameters whenever the
architecture is modified. In this work we instead focus on growing the
architecture without requiring costly retraining. We present a method that adds
new neurons during training without impacting what is already learned, while
improving the training dynamics. We achieve the latter by maximizing the
gradients of the new weights and find the optimal initialization efficiently by
means of the singular value decomposition (SVD). We call this technique
Gradient Maximizing Growth (GradMax) and demonstrate its effectiveness in
variety of vision tasks and architectures.
