Insertion-Deletion Transformer
  We propose the Insertion-Deletion Transformer, a novel transformer-based
neural architecture and training method for sequence generation. The model
consists of two phases that are executed iteratively, 1) an insertion phase and
2) a deletion phase. The insertion phase parameterizes a distribution of
insertions on the current output hypothesis, while the deletion phase
parameterizes a distribution of deletions over the current output hypothesis.
The training method is a principled and simple algorithm, where the deletion
model obtains its signal directly on-policy from the insertion model output. We
demonstrate the effectiveness of our Insertion-Deletion Transformer on
synthetic translation tasks, obtaining significant BLEU score improvement over
an insertion-only model.
