Learning in Gated Neural Networks
  Gating is a key feature in modern neural networks including LSTMs, GRUs and
sparsely-gated deep neural networks. The backbone of such gated networks is a
mixture-of-experts layer, where several experts make regression decisions and
gating controls how to weigh the decisions in an input-dependent manner.
Despite having such a prominent role in both modern and classical machine
learning, very little is understood about parameter recovery of
mixture-of-experts since gradient descent and EM algorithms are known to be
stuck in local optima in such models.
  In this paper, we perform a careful analysis of the optimization landscape
and show that with appropriately designed loss functions, gradient descent can
indeed learn the parameters accurately. A key idea underpinning our results is
the design of two {\em distinct} loss functions, one for recovering the expert
parameters and another for recovering the gating parameters. We demonstrate the
first sample complexity results for parameter recovery in this model for any
algorithm and demonstrate significant performance gains over standard loss
functions in numerical experiments.
