StructFormer: Joint Unsupervised Induction of Dependency and
  Constituency Structure from Masked Language Modeling
  There are two major classes of natural language grammar -- the dependency
grammar that models one-to-one correspondences between words and the
constituency grammar that models the assembly of one or several corresponded
words. While previous unsupervised parsing methods mostly focus on only
inducing one class of grammars, we introduce a novel model, StructFormer, that
can simultaneously induce dependency and constituency structure. To achieve
this, we propose a new parsing framework that can jointly generate a
constituency tree and dependency graph. Then we integrate the induced
dependency relations into the transformer, in a differentiable manner, through
a novel dependency-constrained self-attention mechanism. Experimental results
show that our model can achieve strong results on unsupervised constituency
parsing, unsupervised dependency parsing, and masked language modeling at the
same time.
