How Can We Be So Dense? The Benefits of Using Highly Sparse
  Representations
  Most artificial networks today rely on dense representations, whereas
biological networks rely on sparse representations. In this paper we show how
sparse representations can be more robust to noise and interference, as long as
the underlying dimensionality is sufficiently high. A key intuition that we
develop is that the ratio of the operable volume around a sparse vector divided
by the volume of the representational space decreases exponentially with
dimensionality. We then analyze computationally efficient sparse networks
containing both sparse weights and activations. Simulations on MNIST and the
Google Speech Command Dataset show that such networks demonstrate significantly
improved robustness and stability compared to dense networks, while maintaining
competitive accuracy. We discuss the potential benefits of sparsity on
accuracy, noise robustness, hyperparameter tuning, learning speed,
computational efficiency, and power requirements.
