ViBERTgrid: A Jointly Trained Multi-Modal 2D Document Representation for
  Key Information Extraction from Documents
  Recent grid-based document representations like BERTgrid allow the
simultaneous encoding of the textual and layout information of a document in a
2D feature map so that state-of-the-art image segmentation and/or object
detection models can be straightforwardly leveraged to extract key information
from documents. However, such methods have not achieved comparable performance
to state-of-the-art sequence- and graph-based methods such as LayoutLM and PICK
yet. In this paper, we propose a new multi-modal backbone network by
concatenating a BERTgrid to an intermediate layer of a CNN model, where the
input of CNN is a document image and the BERTgrid is a grid of word embeddings,
to generate a more powerful grid-based document representation, named
ViBERTgrid. Unlike BERTgrid, the parameters of BERT and CNN in our multimodal
backbone network are trained jointly. Our experimental results demonstrate that
this joint training strategy improves significantly the representation ability
of ViBERTgrid. Consequently, our ViBERTgrid-based key information extraction
approach has achieved state-of-the-art performance on real-world datasets.
