On minimal representations of shallow ReLU networks
  The realization function of a shallow ReLU network is a continuous and
piecewise affine function $f:\mathbb R^d\to \mathbb R$, where the domain
$\mathbb R^{d}$ is partitioned by a set of $n$ hyperplanes into cells on which
$f$ is affine. We show that the minimal representation for $f$ uses either $n$,
$n+1$ or $n+2$ neurons and we characterize each of the three cases. In the
particular case, where the input layer is one-dimensional, minimal
representations always use at most $n+1$ neurons but in all higher dimensional
settings there are functions for which $n+2$ neurons are needed. Then we show
that the set of minimal networks representing $f$ forms a
$C^\infty$-submanifold $M$ and we derive the dimension and the number of
connected components of $M$. Additionally, we give a criterion for the
hyperplanes that guarantees that all continuous, piecewise affine functions are
realization functions of appropriate ReLU networks.
