{'fraction_non_alphanumeric': 0.010842779694430755, 'fraction_numerical': 0.0, 'mean_word_length': 5.170212765957447, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On the logic of variable rules

In the introduction we stated two frequently encountered a priori arguments against the variable rule methodology and attempted to refute them. The first rejects the variable rule methodology – and by implication any study of comparable data – on the grounds that variable rules govern token frequencies while generative grammar does not countenance token frequencies. We agree that variable rules are not generative rules of a new sort but an entirely different kind of logical object and that generative grammar indeed does not countenance token frequencies. But we are convinced by empirical work conducted within the variable rule paradigm that token frequencies often display clear patterns and that moreover some knowledge of these patterns forms part of the linguistic abilities of speakers. We conclude that, whatever the drawbacks of the variable rule formalism, studies employing variable rules have shown regularities in linguistic behavior that point to a serious lack in the generative paradigm, narrowly defined. A second line of argument against variable rules which we rejected was based on assumptions about human psychology and about probability theory. The assumption that human beings cannot assess probabilities and behave in accord with them in a natural and unconscious manner appears to be supported by no empirical evidence and does not seem to us plausible a priori. Moreover, experimental evidence to the contrary exists. The argument to probability theory was that a speaker would have to have an internal counting device to keep track of the relative frequencies of linguistic variants that he had heard from his own or other lips in order to behave in accordance with variable rules. But this would require a kind of probability theory that would differ in remarkable and unspecified ways from ordinary probability theory, since the paradigmatic empirical examples of the familiar theory, such as coins, dice, decks of cards, and so on, are not possessed of memories.