Experiments on Properties of Hidden Structures of Sparse Neural Networks
  Sparsity in the structure of Neural Networks can lead to less energy
consumption, less memory usage, faster computation times on convenient
hardware, and automated machine learning. If sparsity gives rise to certain
kinds of structure, it can explain automatically obtained features during
learning.
  We provide insights into experiments in which we show how sparsity can be
achieved through prior initialization, pruning, and during learning, and answer
questions on the relationship between the structure of Neural Networks and
their performance. This includes the first work of inducing priors from network
theory into Recurrent Neural Networks and an architectural performance
prediction during a Neural Architecture Search. Within our experiments, we show
how magnitude class blinded pruning achieves 97.5% on MNIST with 80%
compression and re-training, which is 0.5 points more than without compression,
that magnitude class uniform pruning is significantly inferior to it and how a
genetic search enhanced with performance prediction achieves 82.4% on CIFAR10.
Further, performance prediction for Recurrent Networks learning the Reber
grammar shows an $R^2$ of up to 0.81 given only structural information.
