{'fraction_non_alphanumeric': 0.030261097556534722, 'fraction_numerical': 0.011215906194239103, 'mean_word_length': 4.996069868995633, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A closer look at the performance of neural language models on reflexive anaphor licensing

An emerging line of work uses psycholinguistic methods to evaluate the syntactic generalizations acquired by neural language models (NLMs). While this approach has shown NLMs to be capable of learning a wide range of linguistic knowledge, confounds in the design of previous experiments may have obscured the potential of NLMs to learn certain grammatical phenomena. Here we re-evaluate the performance of a range of NLMs on reflexive anaphor licensing. Under our paradigm, the models consistently show stronger evidence of learning than reported in previous work. Our approach demonstrates the value of well-controlled psycholinguistic methods in gaining a fine-grained understanding of NLM learning potential.1


Introduction
Language models have many practical applications in NLP, while also offering theoretical insights into the learnability of grammatical phenomena. A question of interest to both the NLP and cognitive science communities is what generalizations neural models are able to implicitly acquire while optimizing performance on the language modeling task. To this end, an emerging line of work seeks to evaluate neural language models (NLMs) as "psycholinguistic subjects"that is, assessing the extent to which their probability distributions conform to human judgments on linguistic data. This psycholinguistic assessment is typically done by evaluating the model on minimal pairs of sentences, which differ only at a target word or phrase that toggles the acceptability of the sentence. If an NLM has learned the linguistic phenomenon in question, then we expect it to assign higher probability to sentences that humans judge to be more acceptable. This approach has shown NLMs to be capable of learning some grammatical phenomena (e.g. subject-verb agreement and filler-gap dependencies) while failing on others (Linzen et al., 2016;Lau et al., 2017;Gulordava et al., 2018;Marvin and Linzen, 2018;Tran et al., 2018;.
In evaluating these mixed learning outcomes, we raise a broader question that remains largely unaddressed in the field: What is the standard to which we should be holding artificial language models? An engineering goal within the machine learning community is to build NLMs that approximate human behavior. In this case, an ideal NLM should achieve high performance even on lowfrequency constructions, and the learning signal should be detectable even with coarse experimental paradigms. However, if our scientific goal is to highlight the grammatical phenomena that are possible to learn from sequential data, then we should design our experiments with the aim to give NLMs a fair shot at displaying successful learning.
We demonstrate the value of robust psycholinguistic methods in serving the latter goal by using a new paradigm to re-evaluate the performance of neural language models on English reflexive anaphor licensing (RAL). While previous studies have shown NLMs to fail at RAL in various syntactic configurations Marvin and Linzen, 2018), the complex nature of this dependency highlights the need for well-controlled tests to detect evidence of learning.
A priori, we expect RAL to be difficult to learn for several reasons. From a theoretical perspective, RAL involves the interaction of two syntactic constraints, which may increase the complexity of the representation that needs to be learned (see Section 2.1). In addition, the appearance of a reflexive is never obligatory based on the preceding context -that is, while a reflexive requires an an-tecedent NP licensor, an antecedent NP never requires a reflexive downstream (see Section 2.2).
To take a closer look at previously reported failures of RAL learning, we create a new RAL test suite that controls for confounding variables and uses lexical items that are compatible with small-vocabulary NLMs. Our experiments detect stronger evidence of learning than reported in previous work, demonstrating the value of robust psycholinguistic methods in studying the potential of NLMs to learn complex syntactic phenomena.

Reflexive anaphor licensing (RAL)
English reflexives are licensed only when two different structural constraints are satisfied, which we refer to as LOCALITY and C-COMMAND. These two constraints are independently motivated on theoretical grounds and underlie many syntactic configurations (e.g. Reinhart, 1983;Rizzi, 2013).
First, LOCALITY stipulates that the matching antecedent must be in the same clause as the reflexive. Second, C-COMMAND requires the matching antecedent to be in a c-commanding relation with the reflexive (Reinhart, 1981;Chomsky, 1993). For present purposes, it is sufficient to define c-command as the following: if a node has any sibling node(s) in a syntax tree, then it ccommands its siblings and all of their descendants; if a node has no siblings, then it c-commands everything its parent c-commands.
To illustrate these two constraints, Figure 1 shows the syntax tree for the sentence The fathers said the women near the boys saw themselves. This sentence contains three noun phrases (NPs) that could potentially act as an antecedent for themselves, but only one of them satisfies both structural requirements of RAL: (1) the higher subject NP 1 the fathers c-commands themselves but is not within the local clause, violating LO-CALITY; (2) the lower subject NP 2 the women ccommands themselves locally, licensing the reflexive; (3) the linearly closest NP 3 the boys is within the local clause, but violates C-COMMAND since it is inside a PP modifier. Thus, NP 2 the women is the only possible licensor for the reflexive.
We frame our experiments in terms of the two syntactic constraints involved in RAL, i.e. LO-CALITY and C-COMMAND. This is typically done when testing the linguistic knowledge of humans, in order to probe the nature of linguistic general- izations that are being drawn across different types of constructions. In following this convention, however, we do not intend to claim the NLMs are learning these abstract structural properties per se.

Challenges to learning RAL
Reflexives are never obligatory, in the sense that nothing in the preceding context deterministically predicts an upcoming reflexive. This contrasts with other syntactic dependencies, where the two elements of the dependency mutually require each other. In subject-verb agreement, for example, a subject NP sets the expectation for a downstream verb that agrees in number, and the verb requires a matching subject. This can also be the case for less frequent constructions such as filler-gap dependencies, where the appearance of a filler whword sets the expectation for a gap, and the presence of a gap requires a preceding filler. This property does not hold for reflexive anaphors, as an NP never requires the appearance of a reflexive. Thus, given an upstream reflexive licensor, there is high variance in the downstream contexts. This may make it difficult for NLMs to learn a robust representation for RAL.
Furthermore, although we are interested in reflexive anaphors that occur in an argument position, these pronouns can also occur as an intensifier adjoining right next to an NP, as in The president himself signed my book. Since the intensifier usage does not obey the same structural constraints and thus has a different distribution, it may introduce noise into the learning of RAL.

Paradigms in previous work
Previous work evaluating the ability of neural language models to learn RAL primarily builds upon the paradigms introduced in Marvin and Linzen (2018) and . Both studies conclude that NLMs fail to learn the appropriate licensing condition for reflexives.
In particular, Marvin and Linzen (2018) test whether NLMs learn RAL in relative clauses and sentential complements. Consider the following sample items (1) and (2) from their study: (1) The bankers who the pilot embarrassed hurt *himself / themselves.
In (1), the reflexive himself cannot be licensed by the pilot because the pilot is inside a relative clause, thus violating both LOCALITY and C-COMMAND. In (2), the reflexive themselves is embedded in a sentential complement, so the longdistance subject the bankers cannot license the reflexive since it violates LOCALITY. As is typical in psycholinguistic evaluation of NLMs, previous RAL studies calculate accuracy as the proportion of trials where the model assigns higher probability to the correct reflexive given the prefix, compared to another reflexive that would make the sentence ungrammatical. Since Marvin and Linzen (2018) and  test number and gender agreement, respectively, Marvin and Linzen compare the probability of himself /herself vs. themselves, while Futrell et al. compare the probability of himself vs. herself.
While the failures reported by these studies have been taken as evidence of the limits of NLM learning, they might be attributed to confounding factors in the design of the experiments. As discussed above, previous studies measure accuracy by comparing the probability assigned to different target reflexives given the same context. However, in many standard training corpora, the reflexive pronouns themselves, himself, and herself differ dramatically in frequency, leading to an asymmetry in unigram probabilities (Table 1). This presents a confound, as all models are likely to implicitly factor unigram probabilities when estimating conditional probabilities in context 1 . Thus, even if 1 A unigram frequency is one of the easiest things for a neural model to learn, e.g. as the bias term in the output layer. a model has learned correct generalizations about the relevant features of the context, these generalizations could be obscured by large differences in unigram frequency.
In addition, both Marvin and Linzen (2018) and  use profession nouns that are almost all stereotypically male (e.g. banker, senator). However, many of these nouns occur with low frequency in standard training datasets, so existing materials cannot be used to test RAL learning in models with relatively small vocabularies.
To re-evaluate NLM learning potential of RAL in a more robust way, we use a new paradigm that mitigates the issues raised by unigram probability asymmetries and stereotypically gendered nouns. We describe our methods in Section 3.

Experimental design
As a refinement to existing methods, we evaluate RAL learning in neural language models using well-controlled psycholinguistic experiments. Psycholinguistic evaluation of language models typically measures accuracy as the proportion of trials in which the model correctly assigns higher probabilities to the grammatical sentence in a minimal pair. As discussed above, this difference is affected not only by the expectations set by the context, but also by the unigram probabilities of the target words (in the case of RAL, themselves, himself, and herself ). To avoid this issue, we keep the target reflexive fixed and vary the preceding lexical items in each condition.

Conditions
Each sentence in our test suites has two NPs, a verb, and a target reflexive, as well as material that modulates the syntactic state (e.g. the onset of a relative clause). One NP is in a position that can license a reflexive, and the other NP is not. Our experiments have the following three conditions: 1. Baseline: Both NPs match the number feature of the target reflexive. The sentence is grammatical.
2. Distractor: The NP in the licensing position matches the number of the target, but the other NP mismatches. The sentence is still grammatical, but we have introduced material that may distract the NLM.
3. Ungrammatical: The NP in the licensing position mismatches the number of the target. The sentence is ungrammatical.
We choose to test number instead of gender feature agreement (cf.  because we believe models are more likely to learn a representation of number than gender, as number is more frequently marked than gender in English. There is also evidence of NLMs learning other number-based dependencies such as subject-verb agreement (Linzen et al., 2016).

Evaluation metric
Our accuracy calculation involves a three-way comparison. For a given item, the model makes a correct prediction if the probability of the target reflexive in the Ungrammatical condition is lower than the probability of the target in both the Distractor and Baseline conditions. Accuracy is the proportion of items in the experiment for which the model makes the correct prediction. If the probability of the target is the same across conditions, then the prediction is labeled "correct" with probability 1 /3. Under this measure, chance performance is 33.33%, in contrast to the 50% from existing paradigms that compare grammatical vs. ungrammatical constructions.

Lexical items
Nouns Previous studies on RAL use nouns denoting professions often associated with stereotypical gender, such as lumberjack and hairdresser Marvin and Linzen, 2018). 2 However, these nouns are not inherently gendered, and manipulating the gender of the reflexive does not change the grammaticality of the sentence. Instead, we use high-frequency nouns with lexicalized gender, such as man and woman. This allows us to extend our paradigm to models with smaller vocabularies (see Section 4), for which many profession nouns are out-of-vocabulary (e.g. hairdresser). This also ensures that our experiments can be replicated in the future, as the stereotypical gender of occupations represented in word embeddings can vary across time and cultures (Garg et al., 2018). We selected a total of 12 nouns (6 female and 6 male), with the female and male nouns balanced for frequency of occurrence in the Wikipedia corpus (see Table 1).
Verbs We first manually constructed a list of commonly used reflexive verbs. Using this list, we calculated the relative frequency of their occurrences within a reflexive construction in the Wikipedia corpus, and selected the most frequent ones. We also selected the most frequent verbs by their raw counts in the corpus. A total of 15 verbs were selected using this method.
Counterbalancing To ensure that differences in lexical items in preceding context do not confound the observed effects on the target reflexive, we counterbalance the position of nouns such that each noun occurs both in a licensing and a nonlicensing position equally often. Each noun also appears with each verb equally often.

Logic of experiments
In Experiment 1, we first perform a loose replication of Marvin and Linzen (2018) by adapting their materials into our experimental paradigm. The replication includes relative clause and sentential complement constructions, which we test separately in Experiments 1a and 1b, respectively.
As discussed in Section 2.3, one issue with previous studies is the choice to use lexical items with stereotypical gender. In following experiments, we create new test suites with materials using lexicalized gender. In Experiments 2 and 3, we use our new materials to test relative clause and sentential complement constructions, respectively, for comparison with Marvin and Linzen (2018). Since the relative clause construction tests both LOCAL-ITY and C-COMMAND and the sentential complement construction only tests LOCALITY, we test prepositional phrases in Experiment 4 to isolate the effect of C-COMMAND. Table 2 shows sample items for Experiments 2-4 along with corresponding items from Marvin and Linzen (2018).

Language models
We evaluate RAL in six neural language models, as well as a baseline n-gram model. Together, the models cover a range of vocabulary sizes, architectures, and inductive biases (Table 1). Our goal here is not to draw general conclusions about certain architectures or training regimes, but to present results across a diverse set of models, including those that were previously untestable due to experimental design.  (Chelba et al., 2013). JRNN additionally has convolutional neural network character input embeddings.
Transformer-XL and BERT Next, we test two models based on the Transformer architecture (Vaswani et al., 2017). Since Transformers rely on attention mechanisms, they serve as an interesting comparison against LSTM models. Transformer-XL ("TransXL"; Dai et al., 2019) reuses the hidden states obtained in previous segments, which facilitates modeling of long-term dependencies. BERT (Devlin et al., 2018) is bi-directional, in that it is trained to predict the identity of masked words based on the preceding and following context. 3 Recent work has shown BERT to perform well on reflexive constructions (Goldberg, 2019). Both models were trained on document-level corpora instead of shuffled sentences: WikiText-103 (Merity et al., 2017) for TransXL, and a 3.3-billionword combination of BooksCorpus (Zhu et al., 2015) and Wikipedia for BERT. 3 We use the small, uncased version of BERT (BERTBASE) with no fine-tuning after the initial pre-training tasks.

RNNG and TinyLSTM
The last two neural models in our test suite have identical vocabularies but differing inductive biases: a recurrent neural network grammar ("RNNG"; Dyer et al., 2016) and a vanilla LSTM ("TinyLSTM"). Both models were trained on the 1-million-word English Penn Treebank §2-21 (Marcus et al., 1993), but TinyLSTM is only trained on the terminal word sequences, while RNNG is trained on the full annotations, which contain complete constituency parses. This minimal difference allows us to observe the effect of structural supervision, which has been shown to be beneficial in acquiring certain grammatical dependencies (Kuncoro et al., 2017;Wilcox et al., 2019). Crucially, the vocabulary of these models is too small to acommodate the lexical items used in previous RAL studies.
n-gram As a baseline, we test a 5-gram model trained on the same Wikipedia data as GRNN. We use Kneser-Ney smoothing to perform backoff.

Computing word probabilities
In practice, we calculate accuracy (see Section 3.2) by comparing differentials in log probability space at the target pronoun. To obtain the log probability of word w i assigned by the LSTMs and Transformer models, we compute where h i−1 is the model's hidden state before observing w i . This probability is calculated from the model's softmax activation.
To obtain the log probability of w i in the RNNG, we follow the method used in Hale et al. (2018). We use word-synchronous beam search (Stern et al., 2017) to find the most likely incremental parses, and sum their forward probabilities to approximate P (w 1 , . . . , w i + 1 ) and  P (w 1 , . . . , w i−1 ). We use 100 for the action beam size and 10 for the word beam size.
In contrast to the other models in our test suite, BERT is bi-directional. To obtain the log probability of w i , we first feed BERT a sentence with w i masked out and obtain the word predictions for the masked position. This gives us a probability distribution over words. (2018) First, we perform a loose replication adapting the materials of Marvin and Linzen (2018). Recall that they calculate accuracy as the proportion of trials where the model assigns higher probability to the correct target reflexive given the prefix. Their original materials use profession nouns that are stereotypically male. Since these nouns are out-of-vocabulary for RNNG and TinyLSTM, we run this experiment only on the large-vocabulary models (BERT, TransXL, GRNN, JRNN, 5-gram).

Experiment 1: Marvin and Linzen
Exp. 1a: M&L relative clause We first investigate RAL learning in the relative clause construction. Accuracy scores between our replication and the original study are reported in Table 3 ("RELATIVE CLAUSE CONSTRUCTION"). Under our paradigm, GRNN shows considerable improvement over what was reported in Marvin and Linzen (2018), while the 5-gram model remains at chance. While our metrics are not strictly comparable, the original study reports near-chance accuracy (55% ∼ 50%), while we report accuracy well above chance (70% 33.33%). BERT achieves slightly lower accuracy under our paradigm than was reported in Goldberg (2019) (76% vs. 80%); note, however, that our chance baseline is lower.
Exp. 1b: M&L sentential complement Next, we investigate RAL learning in the sentential complement construction. All neural models perform near ceiling in our paradigm, despite our metric having a lower baseline. GRNN achieves 100% accuracy, showing a marked improvement over previously reported results (Table 3).
Our paradigm allows us to separately probe learning signal for each reflexive, which was not possible in previous studies on RAL. While Table 3 shows high accuracy scores averaged across the pronouns, we ask whether the models achieve different accuracy scores on different pronouns. This cross-pronoun comparison is highlighted in Figure 2. The red bars show the negative log probability differential between the target reflexive in the Ungrammatical and Baseline conditions. The blue bars show the negative log probability differential between the target reflexive in the Distractor and Baseline conditions. If the models learn the correct generalization for RAL, then the red bars should be positive (i.e. above baseline) and higher than the blue bars. : Negative log probability differential at reflexive in relative clause constructions (top, Exp. 1a) and sentential complement constructions (bottom, Exp. 1b) with materials adapted from Marvin and Linzen (2018). Red bars: differential at target reflexive in Ungrammatical vs. Baseline conditions. Blue bars: differential at target reflexive in the Distractor vs. Baseline conditions. If the models learn the correct generalization for RAL, then the red bars should be positive (i.e. above baseline) and higher than the blue bars. Overall, the models exhibit the correct trend for the sentential complement construction, but the pattern is less clear for the relative clause construction. Additionally, herself yields high differentials in the distractor condition as well as the ungrammatical condition, unlike himself and themselves. This may be an artifact of the unbalanced unigram probabilities of the different reflexives.
Overall, the models exhibit the correct trend for the sentential complement construction (bottom), but the pattern is less clear for the relative clause construction (top). Additionally, herself yields high differentials in the distractor condition as well as the ungrammatical condition, unlike himself and themselves. This may be an artifact of the unbalanced unigram probabilities of the different reflexives.
The materials used in Marvin and Linzen (2018) (and our replication, Experiment 1) involve items with stereotypically gendered nouns. This raises two potential issues: (1) gender biases may overshadow number mismatch effects, and (2) the materials can only be used to evaluate models with reasonably large vocabularies. In Experiments 2-4, we create new materials using lexicalized gender rather than stereotypical gender. This allows us to evaluate all seven models in our test suite.

Experiment 2: Relative clause
In Experiment 2, we test RAL learning in the relative clause construction using our new set of materials. Mirroring the structure used in Marvin and Linzen (2018) and Experiment 1a, we attach a rel-ative clause to the subject NP of the sentence. The NP inside the relative clause cannot license the reflexive, as such a relationship would violate both LOCALITY and C-COMMAND.
Accuracy scores are high for most of the large-vocabulary neural models (BERT, TransXL, JRNN) and above chance for GRNN, but at or below chance for the other models (Table 3).

Experiment 3: Sentential complement
In Experiment 3, we test the sentential complement construction using our materials. Recall that the long-distance subject cannot license the reflexive embedded in a sentential complement, because such a relationship would violate LOCALITY (while satisfying C-COMMAND). As shown in Table 2, we place the reflexive inside a complement clause, such that either both c-commanding NPs match the number feature of the reflexive (Baseline), or there is one mismatching NP either in the non-local subject position (Distractor) or the local subject position (Ungrammatical).
All neural models perform near ceiling ( Table 3). The small-vocabulary models RNNG and TinyLSTM show a reduced effect, but RNNG out-   (2019), while the GRNN and 5-gram results marked with ‡ come directly from Marvin and Linzen (2018). These results are also not directly comparable to each other; see Goldberg (2019) and Wolf (2019) for details.

Experiment 4: Prepositional phrase
Since previous studies have focused on the relative clause and sentential complement constructions, C-COMMAND has not been tested separately from LOCALITY. In Experiment 4, we hold LOCALITY constant while manipulating C-COMMAND by placing a distractor NP inside a non-c-commanding PP modifier in the local subject NP. No clausal boundary is introduced. Accuracy scores are reported in the last row of Table 3 ("C-COMMAND ONLY"). Performance is well above chance for all models except TinyLSTM and the 5-gram model. RNNG achieves accuracy comparable to TransXL and shows a clear advantage over TinyLSTM (63% vs. 42%). This suggests that structural supervision can facilitate learning, but large data is ultimately more valuable.

Discussion
In this paper, we used a new paradigm to reevaluate the performance of neural language models on reflexive anaphor licensing. Our methods address issues in previous studies, such as unigram probability asymmetries between target pronouns and the choice to use nouns with stereotypical gender, which may have led to an underestimation of learning signal. The results suggest that NLMs are learning more about RAL than they have previously been given credit for, demonstrating the value of robust psycholinguistic methods in high-lighting the potential of NLMs to learn complex syntactic phenomena.
The value of our approach extends beyond RAL. If we seek to understand the linguistic generalizations that NLMs can potentially acquire, then we must design our experiments to give NLMs a fair shot at displaying successful learning, regardless of the phenomenon under study. In addition, we designed our materials to test LOCAL-ITY and C-COMMAND both separately and simultaneously, motivating NLM research to move beyond testing specific surface constructions to more abstract syntactic constraints (cf. Kuncoro et al.'s (2017) attempt to test for "headedness").
One may argue that this line of research rests upon the assumption that learning a grammar can be identified with learning a language model. Indeed, it is possible that what NLMs learn are not structural aspects characterized in linguistic terms such as LOCALITY and C-COMMAND, but rather properties of the data that are irrelevant to structural considerations. However, we argue that such alternative hypotheses do not necessarily invalidate the approach taken here, as they can be tested in much the same way that our tests so far have been constructed. Furthermore, psycholinguistic evaluation of NLMs allows us not only to engineer human-like models, but also to gain scientific insights into human language acquisition. We advocate that future work in this domain should carefully address these hypotheses about language learning, keeping in mind the co-existing questions that arise from different research agendas.