Generative Flows with Invertible Attentions
  Flow-based generative models have shown an excellent ability to explicitly
learn the probability density function of data via a sequence of invertible
transformations. Yet, learning attentions in generative flows remains
understudied, while it has made breakthroughs in other domains. To fill the
gap, this paper introduces two types of invertible attention mechanisms, i.e.,
map-based and transformer-based attentions, for both unconditional and
conditional generative flows. The key idea is to exploit a masked scheme of
these two attentions to learn long-range data dependencies in the context of
generative flows. The masked scheme allows for invertible attention modules
with tractable Jacobian determinants, enabling its seamless integration at any
positions of the flow-based models. The proposed attention mechanisms lead to
more efficient generative flows, due to their capability of modeling the
long-term data dependencies. Evaluation on multiple image synthesis tasks shows
that the proposed attention flows result in efficient models and compare
favorably against the state-of-the-art unconditional and conditional generative
flows.
