Contextual Temperature for Language Modeling
  Temperature scaling has been widely used as an effective approach to control
the smoothness of a distribution, which helps the model performance in various
tasks. Current practices to apply temperature scaling assume either a fixed, or
a manually-crafted dynamically changing schedule. However, our studies indicate
that the individual optimal trajectory for each class can change with the
context. To this end, we propose contextual temperature, a generalized approach
that learns an optimal temperature trajectory for each vocabulary over the
context. Experimental results confirm that the proposed method significantly
improves state-of-the-art language models, achieving a perplexity of 55.31 and
62.89 on the test set of Penn Treebank and WikiText-2, respectively. In-depth
analyses show that the behaviour of the learned temperature schedules varies
dramatically by vocabulary, and that the optimal schedules help in controlling
the uncertainties. These evidences further justify the need for the proposed
method and its advantages over fixed temperature schedules.
