Network Morphism
  We present in this paper a systematic study on how to morph a well-trained
neural network to a new one so that its network function can be completely
preserved. We define this as \emph{network morphism} in this research. After
morphing a parent network, the child network is expected to inherit the
knowledge from its parent network and also has the potential to continue
growing into a more powerful one with much shortened training time. The first
requirement for this network morphism is its ability to handle diverse morphing
types of networks, including changes of depth, width, kernel size, and even
subnet. To meet this requirement, we first introduce the network morphism
equations, and then develop novel morphing algorithms for all these morphing
types for both classic and convolutional neural networks. The second
requirement for this network morphism is its ability to deal with non-linearity
in a network. We propose a family of parametric-activation functions to
facilitate the morphing of any continuous non-linear activation neurons.
Experimental results on benchmark datasets and typical neural networks
demonstrate the effectiveness of the proposed network morphism scheme.
