Learning Deep ReLU Networks Is Fixed-Parameter Tractable
  We consider the problem of learning an unknown ReLU network with respect to
Gaussian inputs and obtain the first nontrivial results for networks of depth
more than two. We give an algorithm whose running time is a fixed polynomial in
the ambient dimension and some (exponentially large) function of only the
network's parameters.
  Our bounds depend on the number of hidden units, depth, spectral norm of the
weight matrices, and Lipschitz constant of the overall network (we show that
some dependence on the Lipschitz constant is necessary). We also give a bound
that is doubly exponential in the size of the network but is independent of
spectral norm. These results provably cannot be obtained using gradient-based
methods and give the first example of a class of efficiently learnable neural
networks that gradient descent will fail to learn.
  In contrast, prior work for learning networks of depth three or higher
requires exponential time in the ambient dimension, even when the above
parameters are bounded by a constant. Additionally, all prior work for the
depth-two case requires well-conditioned weights and/or positive coefficients
to obtain efficient run-times. Our algorithm does not require these
assumptions.
  Our main technical tool is a type of filtered PCA that can be used to
iteratively recover an approximate basis for the subspace spanned by the hidden
units in the first layer. Our analysis leverages new structural results on
lattice polynomials from tropical geometry.
