F^2-Softmax: Diversifying Neural Text Generation via Frequency
  Factorized Softmax
  Despite recent advances in neural text generation, encoding the rich
diversity in human language remains elusive. We argue that the sub-optimal text
generation is mainly attributable to the imbalanced token distribution, which
particularly misdirects the learning model when trained with the
maximum-likelihood objective. As a simple yet effective remedy, we propose two
novel methods, F^2-Softmax and MefMax, for a balanced training even with the
skewed frequency distribution. MefMax assigns tokens uniquely to frequency
classes, trying to group tokens with similar frequencies and equalize frequency
mass between the classes. F^2-Softmax then decomposes a probability
distribution of the target token into a product of two conditional
probabilities of (i) frequency class, and (ii) token from the target frequency
class. Models learn more uniform probability distributions because they are
confined to subsets of vocabularies. Significant performance gains on seven
relevant metrics suggest the supremacy of our approach in improving not only
the diversity but also the quality of generated texts.
