Dense-to-Sparse Gate for Mixture-of-Experts
  Mixture-of-experts (MoE) is becoming popular due to its success in improving
the model quality, especially in Transformers. By routing tokens with a sparse
gate to a few experts that each only contains part of the full model, MoE keeps
the model size unchanged and significantly reduces per-token computation, which
effectively scales neural networks. However, we found that the current approach
of jointly training experts and the sparse gate introduces a negative impact on
model accuracy, diminishing the efficiency of expensive large-scale model
training. In this work, we proposed Dense-To-Sparse gate (DTS-Gate) for MoE
training. Specifically, instead of using a permanent sparse gate, DTS-Gate
begins as a dense gate that routes tokens to all experts, then gradually and
adaptively becomes sparser while routes to fewer experts. MoE with DTS-Gate
naturally decouples the training of experts and the sparse gate by training all
experts at first and then learning the sparse gate. Experiments show that
compared with the state-of-the-art Switch-Gate in GPT-MoE(1.5B) model with
OpenWebText dataset(40GB), DTS-Gate can obtain 2.0x speed-up to reach the same
validation perplexity, as well as higher FLOPs-efficiency of a 1.42x speed-up.
