Tag, Copy or Predict: A Unified Weakly-Supervised Learning Framework for
  Visual Information Extraction using Sequences
  Visual information extraction (VIE) has attracted increasing attention in
recent years. The existing methods usually first organized optical character
recognition (OCR) results into plain texts and then utilized token-level entity
annotations as supervision to train a sequence tagging model. However, it
expends great annotation costs and may be exposed to label confusion, and the
OCR errors will also significantly affect the final performance. In this paper,
we propose a unified weakly-supervised learning framework called TCPN (Tag,
Copy or Predict Network), which introduces 1) an efficient encoder to
simultaneously model the semantic and layout information in 2D OCR results; 2)
a weakly-supervised training strategy that utilizes only key information
sequences as supervision; and 3) a flexible and switchable decoder which
contains two inference modes: one (Copy or Predict Mode) is to output key
information sequences of different categories by copying a token from the input
or predicting one in each time step, and the other (Tag Mode) is to directly
tag the input sequence in a single forward pass. Our method shows new
state-of-the-art performance on several public benchmarks, which fully proves
its effectiveness.
