Mechanisms of dimensionality reduction and decorrelation in deep neural
  networks
  Deep neural networks are widely used in various domains. However, the nature
of computations at each layer of the deep networks is far from being well
understood. Increasing the interpretability of deep neural networks is thus
important. Here, we construct a mean-field framework to understand how compact
representations are developed across layers, not only in deterministic deep
networks with random weights but also in generative deep networks where an
unsupervised learning is carried out. Our theory shows that the deep
computation implements a dimensionality reduction while maintaining a finite
level of weak correlations between neurons for possible feature extraction.
Mechanisms of dimensionality reduction and decorrelation are unified in the
same framework. This work may pave the way for understanding how a sensory
hierarchy works.
