Deep Neural Networks for Rotation-Invariance Approximation and Learning
  Based on the tree architecture, the objective of this paper is to design deep
neural networks with two or more hidden layers (called deep nets) for
realization of radial functions so as to enable rotational invariance for
near-optimal function approximation in an arbitrarily high dimensional
Euclidian space. It is shown that deep nets have much better performance than
shallow nets (with only one hidden layer) in terms of approximation accuracy
and learning capabilities. In particular, for learning radial functions, it is
shown that near-optimal rate can be achieved by deep nets but not by shallow
nets. Our results illustrate the necessity of depth in neural network design
for realization of rotation-invariance target functions.
