Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  The sparse Mixture-of-Experts (MoE) model is powerful for large-scale
pre-training and has achieved promising results due to its model capacity.
However, with trillions of parameters, MoE is hard to be deployed on cloud or
mobile environment. The inference of MoE requires expert parallelism, which is
not hardware-friendly and communication expensive. Especially for
resource-limited downstream tasks, such sparse structure has to sacrifice a lot
of computing efficiency for limited performance gains. In this work, we observe
most experts contribute scarcely little to the MoE fine-tuning and inference.
We further propose a general method to progressively drop the non-professional
experts for the target downstream task, which preserves the benefits of MoE
while reducing the MoE model into one single-expert dense model. Our
experiments reveal that the fine-tuned single-expert model could preserve 99.3%
benefits from MoE across six different types of tasks while enjoying 2x
inference speed with free communication cost.
