Lossless Acceleration for Seq2seq Generation with Aggressive Decoding
  We study lossless acceleration for seq2seq generation with a novel decoding
algorithm -- Aggressive Decoding. Unlike the previous efforts (e.g.,
non-autoregressive decoding) speeding up seq2seq generation at the cost of
quality loss, our approach aims to yield the identical (or better) generation
compared with autoregressive decoding but in a significant speedup, achieved by
innovative cooperation of aggressive decoding and verification that are both
efficient due to parallel computing.
  We propose two Aggressive Decoding paradigms for 2 kinds of seq2seq tasks: 1)
For the seq2seq tasks whose inputs and outputs are highly similar (e.g.,
Grammatical Error Correction), we propose Input-guided Aggressive Decoding
(IAD) that aggressively copies from the input sentence as drafted decoded
tokens to verify in parallel; 2) For other general seq2seq tasks (e.g., Machine
Translation), we propose Generalized Aggressive Decoding (GAD) that first
employs an additional non-autoregressive decoding model for aggressive decoding
and then verifies in parallel in the autoregressive manner.
  We test Aggressive Decoding on the most popular 6-layer Transformer model on
GPU in multiple seq2seq tasks: 1) For IAD, we show that it can introduce a
7x-9x speedup for the Transformer in Grammatical Error Correction and Text
Simplification tasks with the identical results as greedy decoding; 2) For GAD,
we observe a 3x-5x speedup with the identical or even better quality in two
important seq2seq tasks: Machine Translation and Abstractive Summarization.
Moreover, Aggressive Decoding can benefit even more from stronger computing
devices that are better at parallel computing. Given the lossless quality as
well as significant and promising speedup, we believe Aggressive Decoding may
potentially evolve into a de facto standard for efficient and lossless seq2seq
generation in the near future.
