Gradual DropIn of Layers to Train Very Deep Neural Networks
  We introduce the concept of dynamically growing a neural network during
training. In particular, an untrainable deep network starts as a trainable
shallow network and newly added layers are slowly, organically added during
training, thereby increasing the network's depth. This is accomplished by a new
layer, which we call DropIn. The DropIn layer starts by passing the output from
a previous layer (effectively skipping over the newly added layers), then
increasingly including units from the new layers for both feedforward and
backpropagation. We show that deep networks, which are untrainable with
conventional methods, will converge with DropIn layers interspersed in the
architecture. In addition, we demonstrate that DropIn provides regularization
during training in an analogous way as dropout. Experiments are described with
the MNIST dataset and various expanded LeNet architectures, CIFAR-10 dataset
with its architecture expanded from 3 to 11 layers, and on the ImageNet dataset
with the AlexNet architecture expanded to 13 layers and the VGG 16-layer
architecture.
