Mining the Weights Knowledge for Optimizing Neural Network Structures
  Knowledge embedded in the weights of the artificial neural network can be
used to improve the network structure, such as in network compression. However,
the knowledge is set up by hand, which may not be very accurate, and relevant
information may be overlooked. Inspired by how learning works in the mammalian
brain, we mine the knowledge contained in the weights of the neural network
toward automatic architecture learning in this paper. We introduce a switcher
neural network (SNN) that uses as inputs the weights of a task-specific neural
network (called TNN for short). By mining the knowledge contained in the
weights, the SNN outputs scaling factors for turning off and weighting neurons
in the TNN. To optimize the structure and the parameters of TNN simultaneously,
the SNN and TNN are learned alternately under the same performance evaluation
of TNN using stochastic gradient descent. We test our method on widely used
datasets and popular networks in classification applications. In terms of
accuracy, we outperform baseline networks and other structure learning methods
stably and significantly. At the same time, we compress the baseline networks
without introducing any sparse induction mechanism, and our method, in
particular, leads to a lower compression rate when dealing with simpler
baselines or more difficult tasks. These results demonstrate that our method
can produce a more reasonable structure.
