Lossless Speedup of Autoregressive Translation with Generalized
  Aggressive Decoding
  Different from previous work accelerating translation at the cost of quality
loss, we propose Generalized Aggressive Decoding (GAD) -- a novel decoding
paradigm for lossless speedup of autoregressive translation, through the
collaboration of autoregressive and non-autoregressive translation (NAT) of the
Transformer. At each decoding iteration, GAD aggressively decodes a number of
tokens with NAT as a draft and then verifies them in the autoregressive manner,
where only the tokens that pass the verification are accepted as decoded
tokens. GAD can achieve the same results as autoregressive translation but much
more efficiently because both NAT drafting and autoregressive verification
compute in parallel.
  We conduct experiments in four standard WMT benchmarks and confirm that the
vanilla GAD yields exactly the same results as greedy decoding with an around
$3\times$ speedup, and that its variant (GAD++) with an advanced verification
strategy not only outperforms the greedy translation and even achieves the
comparable translation quality with the beam search result, but also further
improves the decoding speed, resulting in an around $5\times$ speedup over
autoregressive translation. Moreover, GAD can be easily generalized for
lossless speedup of other seq2seq tasks like Abstractive Summarization, and
benefit more from stronger computing devices, demonstrating its potential to
become a de facto decoding paradigm in the future. Our models and codes are
available at https://github.com/hemingkx/GAD.
