Sigsoftmax: Reanalysis of the Softmax Bottleneck
  Softmax is an output activation function for modeling categorical probability
distributions in many applications of deep learning. However, a recent study
revealed that softmax can be a bottleneck of representational capacity of
neural networks in language modeling (the softmax bottleneck). In this paper,
we propose an output activation function for breaking the softmax bottleneck
without additional parameters. We re-analyze the softmax bottleneck from the
perspective of the output set of log-softmax and identify the cause of the
softmax bottleneck. On the basis of this analysis, we propose sigsoftmax, which
is composed of a multiplication of an exponential function and sigmoid
function. Sigsoftmax can break the softmax bottleneck. The experiments on
language modeling demonstrate that sigsoftmax and mixture of sigsoftmax
outperform softmax and mixture of softmax, respectively.
