Distributionally Robust Language Modeling
  Language models are generally trained on data spanning a wide range of topics
(e.g., news, reviews, fiction), but they might be applied to an a priori
unknown target distribution (e.g., restaurant reviews). In this paper, we first
show that training on text outside the test distribution can degrade test
performance when using standard maximum likelihood (MLE) training. To remedy
this without the knowledge of the test distribution, we propose an approach
which trains a model that performs well over a wide range of potential test
distributions. In particular, we derive a new distributionally robust
optimization (DRO) procedure which minimizes the loss of the model over the
worst-case mixture of topics with sufficient overlap with the training
distribution. Our approach, called topic conditional value at risk (topic
CVaR), obtains a 5.5 point perplexity reduction over MLE when the language
models are trained on a mixture of Yelp reviews and news and tested only on
reviews.
