{'fraction_non_alphanumeric': 0.036222596964586844, 'fraction_numerical': 0.00802698145025295, 'mean_word_length': 5.295541401273885, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning constraints on wh-dependencies by learning how to efficiently represent wh-dependencies: A developmental modeling investigation with Fragment Grammars

It’s hotly contested how children learn constraints on the allowed forms in their language, such as constraints on wh-dependencies (these constraints are sometimes called syntactic islands: Chomsky 1973; Pearl and Sprouse 2013). When learning this knowledge, a prerequisite is knowing how to represent wh-dependencies – constraints can then be hypothesized over these dependency representations. Previous work (Pearl and Sprouse, 2013; Liu et al., 2019) explained disparate sets of syntactic island constraints by assuming different wh-dependency representations, without a unified dependency representation capturing all these constraints. Here, we implement a modeled learner attempting to learn a Fragment Grammar (FG) representation (O’Donnell et al., 2011; O’Donnell, 2015) of whdependencies—a representation comprised of potentially different-sized fragments that combine to form full dependencies—that best accounts for the input while being as compact as possible. In particular, FG implements a theory of efficiency that balances the size of the fragments in the resulting grammar while also maximizing the probability of the dependency structures comprised of these fragments. So, when deciding on the fragments to represent from linguistic input, a learner can choose between smaller fragments of the input that may be reused often in different contexts and larger fragments that can be accessed without building up the structure from smaller pieces. The resulting fragment-based wh-dependency representation can then be used to generate any wh-dependency’s probability on the basis of its fragments, and so predict acceptability patterns for stimuli sets that reveal syntactic island knowledge. We find that the identified FG, learned from a realistic sample of wh-dependencies from English-learning children’s input, can generate the attested acceptability judgment patterns for all syntactic islands previously investigated, highlighting how implicit knowledge of wh-dependency constraints can emerge from trying to learn to efficiently represent wh-dependencies more generally. We additionally compare the FG representation’s performance against baselines inspired by previous proposals, finding that one baseline also yields equivalent performance. We discuss how this baseline is similar to and different from the FG representation.


Introduction
It's hotly contested how children learn constraints on the allowed forms in their language, such as constraints on wh-dependencies (these constraints are sometimes called syntactic islands : Chomsky 1973;Pearl and Sprouse 2013). When learning this knowledge, a prerequisite is knowing how to represent wh-dependencies -constraints can then be hypothesized over these dependency representations. Previous work (Pearl and Sprouse, 2013;Liu et al., 2019) explained disparate sets of syntactic island constraints by assuming different wh-dependency representations, without a unified dependency representation capturing all these constraints. Here, we implement a modeled learner attempting to learn a Fragment Grammar (FG) representation (O'Donnell et al., 2011;O'Donnell, 2015) of whdependencies-a representation comprised of potentially different-sized fragments that combine to form full dependencies-that best accounts for the input while being as compact as possible. In particular, FG implements a theory of efficiency that balances the size of the fragments in the resulting grammar while also maximizing the probability of the dependency structures comprised of these fragments. So, when deciding on the fragments to represent from linguistic input, a learner can choose between smaller fragments of the input that may be reused often in different contexts and larger fragments that can be accessed without building up the structure from smaller pieces. The resulting fragment-based wh-dependency representation can then be used to generate any wh-dependency's probability on the basis of its fragments, and so predict acceptability patterns for stimuli sets that reveal syntactic island knowledge. We find that the identified FG, learned from a realistic sample of wh-dependencies from English-learning children's input, can generate the attested acceptability judg-ment patterns for all syntactic islands previously investigated, highlighting how implicit knowledge of wh-dependency constraints can emerge from trying to learn to efficiently represent wh-dependencies more generally. We additionally compare the FG representation's performance against baselines inspired by previous proposals, finding that one baseline also yields equivalent performance. We discuss how this baseline is similar to and different from the FG representation.

Wh-dependency representation
We assume wh-dependencies are represented as sequences of phrase structure nodes that indicate the path from the gap to the wh-word (Pearl and Sprouse, 2013) (1a)-(1b). However, it's unknown whether the phrasal categories (e.g., CP, VP) in this representation need to be lexically subcategorized. For instance, does the dependency path for a whdependency with claim need to include that the verb is claim (1d) or not (1e)? (1) What did Lily claim that Jack forgot?
]]]? b. phrase-structure nodes in syntactic path: IP-VP-CP-IP-VP c. lexical information for those nodes: IP=past, VP=claim, CP=that, IP=past, VP=forget d. possible representations with lexicallysubcategorized VP claim: IP-VP claim -CP-IP-VP, IP past -VP claim -CP-IP past -VP, IP-VP claim -CP that -IP-VP f orget , ... e. possible representations without lexicallysubcategorized VP claim: IP-VP-CP-IP-VP, IP-VP-CP that -IP-VP, IP past -VP-CP that -IP past -VP f orget , ... Figure 1 illustrates the consequences of lexical subcategorization and the related balance of fragment size mentioned earlier, showing two extremes of how to represent the example dependency path from (1). The leftmost representation uses minimalsized fragments (phrasal-only like IP-VP, and lexicalized like IP past ) that may be reused often because they can appear in many different dependencies. This representation has no lexical subcategorization because the lexical information is separate from the phrasal structure. The rightmost representation uses a maximal-sized fragment (representing the entire dependency with both its phrasal structure and lexical pieces) that will only be reused if this exact dependency occurs. This representation has complete lexical subcategorization because all the lexical information is included in this phrase structure fragment. In terms of maximizing the probability of the dependency, each extreme has its drawbacks: the representation relying on minimalsized fragments requires combining many individual fragments, which can lead to a lower probability even if the individual fragments have higher probabilities; the representation relying on the maximalsized fragment likely has a fairly low probability unless this particular dependency happens to occur very frequently (and even if it does, this won't be true for all dependencies). To maximize the probability of a dependency in general, a better approach is to find some intermediate representation, such as the middle one in Figure 1, that involves some larger phrasal fragments incorporating lexical subcategorization (e.g., IP past -VP), as well as some lexical-only fragments (e.g., VP f orget ). In this example intermediate representation, there is thus a tradeoff between larger fragments that don't have to be built every time from smaller fragments (e.g., IP past -VP from IP past and IP-VP) and smaller, more frequently-reused fragments (e.g., VP f orget ). Of course, there are many possible intermediate representations, and the goal for a learner is to identify the best one that maximizes this tradeoff and so yields high probabilities collectively for the dependencies in the input. the lexical information of CPs was included with the phrasal structure). Previous empirical work by Liu et al. (2019) predicted attested judgment patterns for 14 bridge (e.g, say), factive (e.g., know), and manner-of-speaking (e.g., whisper) verbs-see Figure 2c-in terms of the lexical frequency of the main-clause VP. While Liu et al. didn't explicitly propose a theory of representation, their results are compatible with a representation that lexically subcategorizes main-clause VPs (i.e., only the lexical information of the main verb is included with the phrasal structure). Yet, these are only two of many possible types of hypotheses for how the phrasal structure of wh-dependencies could be represented (i.e., different intermediate representations). Using an FG, we can explore the entire hypothesis space that investigates not only which lexical information should be included (e.g., CPs or main VPs), but also what size fragments are the most efficient for the phrasal structure of the dependency to be built from. Importantly, instead of telling the learner beforehand what phrase structure nodes are lexicalized and what size fragments to use, the learner using FGs infers both on the basis of its input.

Learning efficient representations that underlie wh-dependency constraints
We implement a computational-level modeled learner that attempts to identify an FG encoding the most efficient dependency path representation. The model uses Bayesian inference to identify the best representation. In particular, the modeled learner uses a Metropolis-Hastings-based inference algorithm to find the set of fragments that best explains the input, by yielding a high probability for the dependencies in the input. To identify this FG representation, the modeled learner uses the Metropolis-  Hastings algorithm to iteratively resample a potential FG representation for each item in the input and then accepts or rejects the representation to increase the probability of the input data.
To approximate the wh-dependency input that children learn from, we collected 12,704 whdependencies from the CHILDES Treebank (Pearl and Sprouse, 2013) and extracted the dependency path from each. 1 We then estimated the counts of the dependencies that children would encounter by four years old, when some syntactic island knowledge seems to be present (De Villiers et al., 2008). 2 From this input, the modeled learner infers (Figure 2b and d). 3 Notably, the FG representation's fragments lexically subcategorize phrasal structures only for some more-frequent items (e.g., VP think , CP that , VP say -CP). This means the modeled learner automatically determined the best frequency threshold for lexically subcategorizing each individual phrase structure type, due to the goal of efficient representation.

Comparison representations
We compared the FG representation's performance against several trigram-based baseline representations (2), all of which used the same input as the FG model. 4 We chose trigram-based representations, as n-grams are common representations in language modeling (see Manning and Schutze 1999 for a review), and trigram-based representations have been used in prior successful models that predict adult judgement patterns of wh-dependencies (Pearl and Sprouse, 2013). A trigram-based representation also can (i) be paired with a straightforward learning algorithm (e.g., tracking frequencies of the trigrams in the input), and (ii) can transparently reflect different proposals for lexical subcategorization, as in (2).
(2) Baseline trigam representations a. no-lexicalization: phrase labels only, e.g., "IP-VP-CP" b. fully-lexicalized: subcategorized phrase labels, e.g., "IP past -VP claim -CP that " c. CP-lexicalized (from Pearl and Sprouse 2013): only CP is subcategorized, e.g., "IP-VP-CP that " d. main-V-lexicalized (in line with Liu et al. 2019): only main V is subcategorized, e.g., "IP-VP claim -CP" We selected the no-lexicalization and the fullylexicalized representations as the two extremes of our hypothesis space; we can include no lexical information or all the lexical information for phrase structure nodes in a trigram-based dependency representation. The remaining two representations each implement a hypothesis about what lexical information should be included in the phrasal structure, inspired by previous work: the CP-lexicalized representation from Pearl and Sprouse (2013), and the main-V-lexicalized representation from Liu et al. (2019).
Most baselines failed to capture the full range of acceptability judgment patterns: the no-lexicalized failed to capture Adjunct and Whether islands, as well as the verb frequency effect; the fullylexicalized failed to capture Adjunct islands; and the CP-lexicalized failed to capture the verb frequency effect. However, the main-V-lexicalized did capture all the acceptability patterns. We note that the FG representation also lexicalized main verbs (though only those that were more frequent), and so has this in common with the main-V-lexicalized baseline (which lexicalized all main verbs, irrespective of frequency). We note that one advantage of the inferred FG representation over the main-Vlexicalized representation is that the FG representation was automatically learned -including which parts are lexicalized and how large the pieces are that comprise a dependency path -rather than needing to be specified beforehand, as the trigram-based main-V-lexicalized baseline was.

Conclusion
Here we have explored how children could learn constraints on English wh-dependencies by focusing their learning efforts on how to efficiently represent wh-dependencies, rather than trying to explicitly learn the constraints. The specific approach we explored involved a modeled learner attempting to identify the best Fragment Grammar (FG) for efficiently representing the wh-dependencies encountered in English child-directed speech. The FG representation allowed the modeled learner to generate all the acceptability judgement patterns previously attested to reflect knowledge of different constraints on wh-dependencies, known as syntactic islands. Because the modeled learner learned from input that four-year-olds would encounter, one testable prediction that future behavioral work can investigate is that four-year-olds should in fact have acquired all the syntactic knowledge assessed via the acceptability judgment patterns used here if four-year-olds are in fact using an FG representation. Additionally, future work can investigate predictions for other wh-dependency constraints known to be acquired by children around age four (De Villiers et al., 2008), comparing the FG representation against other representational possibilities, such as the main-V-lexicalized baseline. 223