Training and Inference on Any-Order Autoregressive Models the Right Way
  Conditional inference on arbitrary subsets of variables is a core problem in
probabilistic inference with important applications such as masked language
modeling and image inpainting. In recent years, the family of Any-Order
Autoregressive Models (AO-ARMs) -- which includes popular models such as XLNet
-- has shown breakthrough performance in arbitrary conditional tasks across a
sweeping range of domains. But, in spite of their success, in this paper we
identify significant improvements to be made to previous formulations of
AO-ARMs. First, we show that AO-ARMs suffer from redundancy in their
probabilistic model, i.e., they define the same distribution in multiple
different ways. We alleviate this redundancy by training on a smaller set of
univariate conditionals that still maintains support for efficient arbitrary
conditional inference. Second, we upweight the training loss for univariate
conditionals that are evaluated more frequently during inference. Our method
leads to improved performance with no compromises on tractability, giving
state-of-the-art likelihoods in arbitrary conditional modeling on text (Text8),
image (CIFAR10, ImageNet32), and continuous tabular data domains.
