Depth separation beyond radial functions
  High-dimensional depth separation results for neural networks show that
certain functions can be efficiently approximated by two-hidden-layer networks
but not by one-hidden-layer ones in high-dimensions $d$. Existing results of
this type mainly focus on functions with an underlying radial or
one-dimensional structure, which are usually not encountered in practice. The
first contribution of this paper is to extend such results to a more general
class of functions, namely functions with piece-wise oscillatory structure, by
building on the proof strategy of (Eldan and Shamir, 2016). We complement these
results by showing that, if the domain radius and the rate of oscillation of
the objective function are constant, then approximation by one-hidden-layer
networks holds at a $\mathrm{poly}(d)$ rate for any fixed error threshold.
  A common theme in the proofs of depth-separation results is the fact that
one-hidden-layer networks fail to approximate high-energy functions whose
Fourier representation is spread in the domain. On the other hand, existing
approximation results of a function by one-hidden-layer neural networks rely on
the function having a sparse Fourier representation. The choice of the domain
also represents a source of gaps between upper and lower approximation bounds.
Focusing on a fixed approximation domain, namely the sphere $\mathbb{S}^{d-1}$
in dimension $d$, we provide a characterisation of both functions which are
efficiently approximable by one-hidden-layer networks and of functions which
are provably not, in terms of their Fourier expansion.
