The Limitations of Limited Context for Constituency Parsing
  Incorporating syntax into neural approaches in NLP has a multitude of
practical and scientific benefits. For instance, a language model that is
syntax-aware is likely to be able to produce better samples; even a
discriminative model like BERT with a syntax module could be used for core NLP
tasks like unsupervised syntactic parsing. Rapid progress in recent years was
arguably spurred on by the empirical success of the Parsing-Reading-Predict
architecture of (Shen et al., 2018a), later simplified by the Order Neuron LSTM
of (Shen et al., 2019). Most notably, this is the first time neural approaches
were able to successfully perform unsupervised syntactic parsing (evaluated by
various metrics like F-1 score).
  However, even heuristic (much less fully mathematical) understanding of why
and when these architectures work is lagging severely behind. In this work, we
answer representational questions raised by the architectures in (Shen et al.,
2018a, 2019), as well as some transition-based syntax-aware language models
(Dyer et al., 2016): what kind of syntactic structure can current neural
approaches to syntax represent? Concretely, we ground this question in the
sandbox of probabilistic context-free-grammars (PCFGs), and identify a key
aspect of the representational power of these approaches: the amount and
directionality of context that the predictor has access to when forced to make
parsing decision. We show that with limited context (either bounded, or
unidirectional), there are PCFGs, for which these approaches cannot represent
the max-likelihood parse; conversely, if the context is unlimited, they can
represent the max-likelihood parse of any PCFG.
