Tighter Sparse Approximation Bounds for ReLU Neural Networks
  A well-known line of work (Barron, 1993; Breiman, 1993; Klusowski & Barron,
2018) provides bounds on the width $n$ of a ReLU two-layer neural network
needed to approximate a function $f$ over the ball
$\mathcal{B}_R(\mathbb{R}^d)$ up to error $\epsilon$, when the Fourier based
quantity $C_f = \frac{1}{(2\pi)^{d/2}} \int_{\mathbb{R}^d} \|\xi\|^2
|\hat{f}(\xi)| \ d\xi$ is finite. More recently Ongie et al. (2019) used the
Radon transform as a tool for analysis of infinite-width ReLU two-layer
networks. In particular, they introduce the concept of Radon-based
$\mathcal{R}$-norms and show that a function defined on $\mathbb{R}^d$ can be
represented as an infinite-width two-layer neural network if and only if its
$\mathcal{R}$-norm is finite. In this work, we extend the framework of Ongie et
al. (2019) and define similar Radon-based semi-norms ($\mathcal{R},
\mathcal{U}$-norms) such that a function admits an infinite-width neural
network representation on a bounded open set $\mathcal{U} \subseteq
\mathbb{R}^d$ when its $\mathcal{R}, \mathcal{U}$-norm is finite. Building on
this, we derive sparse (finite-width) neural network approximation bounds that
refine those of Breiman (1993); Klusowski & Barron (2018). Finally, we show
that infinite-width neural network representations on bounded open sets are not
unique and study their structure, providing a functional view of mode
connectivity.
