Neural Network Layer Algebra: A Framework to Measure Capacity and
  Compression in Deep Learning
  We present a new framework to measure the intrinsic properties of (deep)
neural networks. While we focus on convolutional networks, our framework can be
extrapolated to any network architecture. In particular, we evaluate two
network properties, namely, capacity, which is related to expressivity, and
compression, which is related to learnability. Both these properties depend
only on the network structure and are independent of the network parameters. To
this end, we propose two metrics: the first one, called layer complexity,
captures the architectural complexity of any network layer; and, the second
one, called layer intrinsic power, encodes how data is compressed along the
network. The metrics are based on the concept of layer algebra, which is also
introduced in this paper. This concept is based on the idea that the global
properties depend on the network topology, and the leaf nodes of any neural
network can be approximated using local transfer functions, thereby, allowing a
simple computation of the global metrics. We show that our global complexity
metric can be calculated and represented more conveniently than the widely-used
VC dimension. We also compare the properties of various state-of-the art
architectures using our metrics and use the properties to analyze their
accuracy on benchmark image classification datasets.
