Correcting diacritics and typos with a ByT5 transformer model
  Due to the fast pace of life and online communications and the prevalence of
English and the QWERTY keyboard, people tend to forgo using diacritics, make
typographical errors (typos) when typing in other languages. Restoring
diacritics and correcting spelling is important for proper language use and the
disambiguation of texts for both humans and downstream algorithms. However,
both of these problems are typically addressed separately: the state-of-the-art
diacritics restoration methods do not tolerate other typos, but classical
spellcheckers also cannot deal adequately with all the diacritics missing. In
this work, we tackle both problems at once by employing the newly-developed
universal ByT5 byte-level seq2seq transformer model that requires no
language-specific model structures. For a comparison, we perform diacritics
restoration on benchmark datasets of 12 languages, with the addition of
Lithuanian. The experimental investigation proves that our approach is able to
achieve results (> 98%) comparable to the previous state-of-the-art, despite
being trained less and on fewer data. Our approach is also able to restore
diacritics in words not seen during training with > 76% accuracy. Our
simultaneous diacritics restoration and typos correction approach reaches > 94%
alpha-word accuracy on the 13 languages. It has no direct competitors and
strongly outperforms classical spell-checking or dictionary-based approaches.
We also demonstrate all the accuracies to further improve with more training.
Taken together, this shows the great real-world application potential of our
suggested methods to more data, languages, and error classes.
