TraDE: Transformers for Density Estimation
  We present TraDE, a self-attention-based architecture for auto-regressive
density estimation with continuous and discrete valued data. Our model is
trained using a penalized maximum likelihood objective, which ensures that
samples from the density estimate resemble the training data distribution. The
use of self-attention means that the model need not retain conditional
sufficient statistics during the auto-regressive process beyond what is needed
for each covariate. On standard tabular and image data benchmarks, TraDE
produces significantly better density estimates than existing approaches such
as normalizing flow estimators and recurrent auto-regressive models. However
log-likelihood on held-out data only partially reflects how useful these
estimates are in real-world applications. In order to systematically evaluate
density estimators, we present a suite of tasks such as regression using
generated samples, out-of-distribution detection, and robustness to noise in
the training data and demonstrate that TraDE works well in these scenarios.
