Parallel Iterative Edit Models for Local Sequence Transduction
  We present a Parallel Iterative Edit (PIE) model for the problem of local
sequence transduction arising in tasks like Grammatical error correction (GEC).
Recent approaches are based on the popular encoder-decoder (ED) model for
sequence to sequence learning. The ED model auto-regressively captures full
dependency among output tokens but is slow due to sequential decoding. The PIE
model does parallel decoding, giving up the advantage of modelling full
dependency in the output, yet it achieves accuracy competitive with the ED
model for four reasons: 1.~predicting edits instead of tokens, 2.~labeling
sequences instead of generating sequences, 3.~iteratively refining predictions
to capture dependencies, and 4.~factorizing logits over edits and their token
argument to harness pre-trained language models like BERT. Experiments on tasks
spanning GEC, OCR correction and spell correction demonstrate that the PIE
model is an accurate and significantly faster alternative for local sequence
transduction.
