The Importance of Generation Order in Language Modeling
  Neural language models are a critical component of state-of-the-art systems
for machine translation, summarization, audio transcription, and other tasks.
These language models are almost universally autoregressive in nature,
generating sentences one token at a time from left to right. This paper studies
the influence of token generation order on model quality via a novel two-pass
language model that produces partially-filled sentence "templates" and then
fills in missing tokens. We compare various strategies for structuring these
two passes and observe a surprisingly large variation in model quality. We find
the most effective strategy generates function words in the first pass followed
by content words in the second. We believe these experimental results justify a
more extensive investigation of generation order for neural language models.
