Representational Capacity of Deep Neural Networks -- A Computing Study
  There is some theoretical evidence that deep neural networks with multiple
hidden layers have a potential for more efficient representation of
multidimensional mappings than shallow networks with a single hidden layer. The
question is whether it is possible to exploit this theoretical advantage for
finding such representations with help of numerical training methods. Tests
using prototypical problems with a known mean square minimum did not confirm
this hypothesis. Minima found with the help of deep networks have always been
worse than those found using shallow networks. This does not directly
contradict the theoretical findings---it is possible that the superior
representational capacity of deep networks is genuine while finding the mean
square minimum of such deep networks is a substantially harder problem than
with shallow ones.
