Improving Neural Network with Uniform Sparse Connectivity
  Neural network forms the foundation of deep learning and numerous AI
applications. Classical neural networks are fully connected, expensive to train
and prone to overfitting. Sparse networks tend to have convoluted structure
search, suboptimal performance and limited usage. We proposed the novel uniform
sparse network (USN) with even and sparse connectivity within each layer. USN
has one striking property that its performance is independent of the
substantial topology variation and enormous model space, thus offers a
search-free solution to all above mentioned issues of neural networks. USN
consistently and substantially outperforms the state-of-the-art sparse network
models in prediction accuracy, speed and robustness. It even achieves higher
prediction accuracy than the fully connected network with only 0.55% parameters
and 1/4 computing time and resources. Importantly, USN is conceptually simple
as a natural generalization of fully connected network with multiple
improvements in accuracy, robustness and scalability. USN can replace the
latter in a range of applications, data types and deep learning architectures.
We have made USN open source at https://github.com/datapplab/sparsenet.
