Text Detoxification using Large Pre-trained Neural Models
  We present two novel unsupervised methods for eliminating toxicity in text.
Our first method combines two recent ideas: (1) guidance of the generation
process with small style-conditional language models and (2) use of
paraphrasing models to perform style transfer. We use a well-performing
paraphraser guided by style-trained language models to keep the text content
and remove toxicity. Our second method uses BERT to replace toxic words with
their non-offensive synonyms. We make the method more flexible by enabling BERT
to replace mask tokens with a variable number of words. Finally, we present the
first large-scale comparative study of style transfer models on the task of
toxicity removal. We compare our models with a number of methods for style
transfer. The models are evaluated in a reference-free way using a combination
of unsupervised style transfer metrics. Both methods we suggest yield new SOTA
results.
