The Monte Carlo Transformer: a stochastic self-attention model for
  sequence prediction
  This paper introduces the Sequential Monte Carlo Transformer, an original
approach that naturally captures the observations distribution in a transformer
architecture. The keys, queries, values and attention vectors of the network
are considered as the unobserved stochastic states of its hidden structure.
This generative model is such that at each time step the received observation
is a random function of its past states in a given attention window. In this
general state-space setting, we use Sequential Monte Carlo methods to
approximate the posterior distributions of the states given the observations,
and to estimate the gradient of the log-likelihood. We hence propose a
generative model giving a predictive distribution, instead of a single-point
estimate.
