PixelSNAIL: An Improved Autoregressive Generative Model
  Autoregressive generative models consistently achieve the best results in
density estimation tasks involving high dimensional data, such as images or
audio. They pose density estimation as a sequence modeling task, where a
recurrent neural network (RNN) models the conditional distribution over the
next element conditioned on all previous elements. In this paradigm, the
bottleneck is the extent to which the RNN can model long-range dependencies,
and the most successful approaches rely on causal convolutions, which offer
better access to earlier parts of the sequence than conventional RNNs. Taking
inspiration from recent work in meta reinforcement learning, where dealing with
long-range dependencies is also essential, we introduce a new generative model
architecture that combines causal convolutions with self attention. In this
note, we describe the resulting model and present state-of-the-art
log-likelihood results on CIFAR-10 (2.85 bits per dim) and $32 \times 32$
ImageNet (3.80 bits per dim). Our implementation is available at
https://github.com/neocxi/pixelsnail-public
