An initial alignment between neural network and target is needed for
  gradient descent to learn
  This paper introduces the notion of "Initial Alignment" (INAL) between a
neural network at initialization and a target function. It is proved that if a
network and target function do not have a noticeable INAL, then noisy gradient
descent on a fully connected network with normalized i.i.d. initialization will
not learn in polynomial time. Thus a certain amount of knowledge about the
target (measured by the INAL) is needed in the architecture design. This also
provides an answer to an open problem posed in [AS20]. The results are based on
deriving lower-bounds for descent algorithms on symmetric neural networks
without explicit knowledge of the target function beyond its INAL.
