Nonparametric Learning of Two-Layer ReLU Residual Units
  We describe an algorithm that learns two-layer residual units with rectified
linear unit (ReLU) activation: suppose the input $\mathbf{x}$ is from a
distribution with support space $\mathbb{R}^d$ and the ground-truth generative
model is such a residual unit, given by \[\mathbf{y}=
\boldsymbol{B}^\ast\left[\left(\boldsymbol{A}^\ast\mathbf{x}\right)^+ +
\mathbf{x}\right]\text{,}\] where ground-truth network parameters
$\boldsymbol{A}^\ast \in \mathbb{R}^{d\times d}$ is a nonnegative full-rank
matrix and $\boldsymbol{B}^\ast \in \mathbb{R}^{m\times d}$ is full-rank with
$m \geq d$ and for $\mathbf{c} \in \mathbb{R}^d$, $[\mathbf{c}^{+}]_i =
\max\{0, c_i\}$. We design layer-wise objectives as functionals whose analytic
minimizers express the exact ground-truth network in terms of its parameters
and nonlinearities. Following this objective landscape, learning residual units
from finite samples can be formulated using convex optimization of a
nonparametric function: for each layer, we first formulate the corresponding
empirical risk minimization (ERM) as a positive semi-definite quadratic program
(QP), then we show the solution space of the QP can be equivalently determined
by a set of linear inequalities, which can then be efficiently solved by linear
programming (LP). We further prove the statistical strong consistency of our
algorithm, and demonstrate the robustness and sample efficiency of our
algorithm by experiments.
