Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient
  Algorithms
  Mixture-of-Experts (MoE) is a widely popular model for ensemble learning and
is a basic building block of highly successful modern neural networks as well
as a component in Gated Recurrent Units (GRU) and Attention networks. However,
present algorithms for learning MoE including the EM algorithm, and gradient
descent are known to get stuck in local optima. From a theoretical viewpoint,
finding an efficient and provably consistent algorithm to learn the parameters
remains a long standing open problem for more than two decades. In this paper,
we introduce the first algorithm that learns the true parameters of a MoE model
for a wide class of non-linearities with global consistency guarantees. While
existing algorithms jointly or iteratively estimate the expert parameters and
the gating paramters in the MoE, we propose a novel algorithm that breaks the
deadlock and can directly estimate the expert parameters by sensing its echo in
a carefully designed cross-moment tensor between the inputs and the output.
Once the experts are known, the recovery of gating parameters still requires an
EM algorithm; however, we show that the EM algorithm for this simplified
problem, unlike the joint EM algorithm, converges to the true parameters. We
empirically validate our algorithm on both the synthetic and real data sets in
a variety of settings, and show superior performance to standard baselines.
