Spelling Error Correction with Soft-Masked BERT
  Spelling error correction is an important yet challenging task because a
satisfactory solution of it essentially needs human-level language
understanding ability. Without loss of generality we consider Chinese spelling
error correction (CSC) in this paper. A state-of-the-art method for the task
selects a character from a list of candidates for correction (including
non-correction) at each position of the sentence on the basis of BERT, the
language representation model. The accuracy of the method can be sub-optimal,
however, because BERT does not have sufficient capability to detect whether
there is an error at each position, apparently due to the way of pre-training
it using mask language modeling. In this work, we propose a novel neural
architecture to address the aforementioned issue, which consists of a network
for error detection and a network for error correction based on BERT, with the
former being connected to the latter with what we call soft-masking technique.
Our method of using `Soft-Masked BERT' is general, and it may be employed in
other language detection-correction problems. Experimental results on two
datasets demonstrate that the performance of our proposed method is
significantly better than the baselines including the one solely based on BERT.
