Lacuna Reconstruction: Self-supervised Pre-training for Low-Resource
  Historical Document Transcription
  We present a self-supervised pre-training approach for learning rich visual
language representations for both handwritten and printed historical document
transcription. After supervised fine-tuning of our pre-trained encoder
representations for low-resource document transcription on two languages, (1) a
heterogeneous set of handwritten Islamicate manuscript images and (2) early
modern English printed documents, we show a meaningful improvement in
recognition accuracy over the same supervised model trained from scratch with
as few as 30 line image transcriptions for training. Our masked language
model-style pre-training strategy, where the model is trained to be able to
identify the true masked visual representation from distractors sampled from
within the same line, encourages learning robust contextualized language
representations invariant to scribal writing style and printing noise present
across documents.
