{'fraction_non_alphanumeric': 0.02197802197802198, 'fraction_numerical': 0.001332001332001332, 'mean_word_length': 4.58364312267658, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The evaluation metric in generative grammar

The subject which I would like to treat in this paper is the evaluation metric in generative grammar. Why? Arguably, the evaluation metric is both the most novel and the most important concept in the development of generative grammar by Noam Chomsky. And yet it is at the same time one of the least recognized and surely most misunderstood of the core concepts of generative grammar. So there you are: the evaluation metric is critically important, it is arguably novel, it is misunderstood, and at some times and in some places, it has even been reviled. What better reasons could there be for spending our time today talking about it? I would like, first, to explain the idea of the evaluation metric in early generative grammar; this will mean exploring the separate ideas of (1) a prior over the set of grammars and (2) a measure of goodness of fit to the data. Second, I will very briefly trace how those two ideas have been developed in the world of machine learning over the last few decades, and as you will see, the picture I see is one of lost opportunity for us linguists, though the opportunities remain open to us. To repeat, then: The notion of an evaluation metric played a central role in the first decades of generative grammar. The idea of an evaluation metric was tightly bound up with the notion of explanatory adequacy. I am going to sketch an idealized picture of what I take it that Noam Chomsky proposed in his early work, The Logical Structure of Linguistic Theory [notably in Chapters 2 and 4 there] and Syntatic Structures. First, a grammar can be thought of as a scientific theory of a language, and any grammar’s predictions can be tested against real data (whether corpus-based or judgment-based), and in most cases we can compare any pair of grammars on the basis of linguistic data. If that were the whole story, though, we would not need an evaluation metric. But there is more. Early on in the lifespan of the enterprise, we linguists learn about the formal properties of the very best grammars, and as we do so, we develop a higher-order description language, a language in which grammars are written; and as we do this, we will ensure that those phenomena which are common across grammars can be expressed in a compact way, so that within a brief period of disciplinary evolution, a metalanguage will emerge in which compactness of expression will fit well with what is found in a wide range of languages. Out of this process will naturally emerge a body of meta-knowledge about the nature of language, and this body of meta-knowledge lies simply in the way in which grammars are written. This conception of the scientific character of linguistic research—of linguistics as a science—was developed explicitly to counter three tendencies that were present or in any event that seemed to be present, either explicitly or just below the surface, in the work of a number of linguists working in the Sapir-Bloomfield tradition: