Information Scaling Law of Deep Neural Networks
  With the rapid development of Deep Neural Networks (DNNs), various network
models that show strong computing power and impressive expressive power are
proposed. However, there is no comprehensive informational interpretation of
DNNs from the perspective of information theory. Due to the nonlinear function
and the uncertain number of layers and neural units used in the DNNs, the
network structure shows nonlinearity and complexity. With the typical DNNs
named Convolutional Arithmetic Circuits (ConvACs), the complex DNNs can be
converted into mathematical formula. Thus, we can use rigorous mathematical
theory especially the information theory to analyse the complicated DNNs. In
this paper, we propose a novel information scaling law scheme that can
interpret the network's inner organization by information theory. First, we
show the informational interpretation of the activation function. Secondly, we
prove that the information entropy increases when the information is
transmitted through the ConvACs. Finally, we propose the information scaling
law of ConvACs through making a reasonable assumption.
