LIMIT-BERT : Linguistic Informed Multi-Task BERT
  In this paper, we present a Linguistic Informed Multi-Task BERT (LIMIT-BERT)
for learning language representations across multiple linguistic tasks by
Multi-Task Learning (MTL). LIMIT-BERT includes five key linguistic syntax and
semantics tasks: Part-Of-Speech (POS) tags, constituent and dependency
syntactic parsing, span and dependency semantic role labeling (SRL). Besides,
LIMIT-BERT adopts linguistics mask strategy: Syntactic and Semantic Phrase
Masking which mask all of the tokens corresponding to a syntactic/semantic
phrase. Different from recent Multi-Task Deep Neural Networks (MT-DNN) (Liu et
al., 2019), our LIMIT-BERT is linguistically motivated and learning in a
semi-supervised method which provides large amounts of linguistic-task data as
same as BERT learning corpus. As a result, LIMIT-BERT not only improves
linguistic tasks performance but also benefits from a regularization effect and
linguistic information that leads to more general representations to help adapt
to new tasks and domains. LIMIT-BERT obtains new state-of-the-art or
competitive results on both span and dependency semantic parsing on Propbank
benchmarks and both dependency and constituent syntactic parsing on Penn
Treebank.
