Separation of Scales and a Thermodynamic Description of Feature Learning
  in Some CNNs
  Deep neural networks (DNNs) are powerful tools for compressing and distilling
information. Their scale and complexity, often involving billions of
inter-dependent internal degrees of freedom, renders direct microscopic
analysis difficult. Under such circumstances, a common strategy is to identify
slow degrees of freedom that average out the erratic behavior of the underlying
fast microscopic variables. Here, we identify such a separation of scales
occurring in fully trained over-parameterized deep convolutional neural
networks (CNNs). Specifically, we show that DNN layers couple only through the
second moment (kernels) of their activations and pre-activations. Moreover, in
various settings, the latter fluctuate in a nearly Gaussian manner. For CNNs
with infinitely many channels, these kernels are inert, while for finite CNNs
they adapt to the data. In several deep non-linear CNN models trained on real
data, the resulting thermodynamic theory of deep learning yields accurate
predictions. In addition, it provides new ways of analyzing and understanding
CNNs, and DNNs in general.
