Position Masking for Language Models
  Masked language modeling (MLM) pre-training models such as BERT corrupt the
input by replacing some tokens with [MASK] and then train a model to
reconstruct the original tokens. This is an effective technique which has led
to good results on all NLP benchmarks. We propose to expand upon this idea by
masking the positions of some tokens along with the masked input token ids. We
follow the same standard approach as BERT masking a percentage of the tokens
positions and then predicting their original values using an additional fully
connected classifier stage. This approach has shown good performance gains
(.3\% improvement) for the SQUAD additional improvement in convergence times.
For the Graphcore IPU the convergence of BERT Base with position masking
requires only 50\% of the tokens from the original BERT paper.
