Doubly Sparse: Sparse Mixture of Sparse Experts for Efficient Softmax
  Inference
  Computations for the softmax function are significantly expensive when the
number of output classes is large. In this paper, we present a novel softmax
inference speedup method, Doubly Sparse Softmax (DS-Softmax), that leverages
sparse mixture of sparse experts to efficiently retrieve top-k classes.
Different from most existing methods that require and approximate a fixed
softmax, our method is learning-based and can adapt softmax weights for a
better inference speedup. In particular, our method learns a two-level
hierarchy which divides entire output class space into several partially
overlapping experts. Each expert is sparse and only contains a subset of output
classes. To find top-k classes, a sparse mixture enables us to find the most
probable expert quickly, and the sparse expert enables us to search within a
small-scale softmax. We empirically conduct evaluation on several real-world
tasks, including neural machine translation, language modeling and image
classification, and demonstrate that significant computation reductions can be
achieved at no performance loss.
