{'fraction_non_alphanumeric': 0.030456852791878174, 'fraction_numerical': 0.015228426395939087, 'mean_word_length': 5.143884892086331, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Linguistic Fundamentals for Natural Language Processing II: 100 Essentials from Semantics and Pragmatics

Semantics is commonly defined as “the study of meaning” and pragmatics is generally referred to as “the study of meaning in context”. In other words, semantics deal with the sentence meaning (e.g., literal meaning of “How are you?”), whereas pragmatics is more concerned with the speaker/utterance meaning (e.g., the greeting meaning of “How are you?”). Both fields interact with each other and with other lower layers of language such as phonology, morphology, and syntax in many ways. Semantics and pragmatics have been established as research fields a long time ago and been widely studied ever since, as also evidenced by a number of introductory text books dating from the 1980s to today (Levinson 1983; Cruse 2000; Kadmon 2001; Birner 2012/2013; Kroeger 2018). Previous introductory books are remarkably rich with linguistic theorems and provide a comprehensive introduction to the topic. However, the majority offer a linguistic perspective that may be hard to understand for young generations of NLP researchers because of the changing dynamics of the NLP field. As the field has been dominated with end-to-end data driven models, the linguistic phenomena that may help NLP researchers in various ways (e.g., more insightful analysis, more sensible evaluation techniques, more informed model designs, more diverse data collection/annotation schemes) have been mostly neglected. This book, authored by Emily Bender and Alex Lascarides, aims to fill this gap and intends to create a common language between linguists and today’s NLP researchers to ease collaboration. The book covers most of the key issues in semantics and pragmatics, ranging from “meaning of words” to “meaning of utterances in dialogue”, including the discourse structure and coherence relations for which not many resources were available. The book contains 14 chapters and 101 essentials/sections. Each chapter is organized as a collection of self-explanatory sections with sentence headers like “#10 There is ambiguity at all levels”,1 such that a summary of the book can easily be generated by concatenating the headers. The book is organized cleverly: Even though the chapters are kept connected throughout the book, most sections can still be grasped when read separately. Plenty of examples are provided in a variety of languages for each concept, leaving the reader with a feeling of appreciation and an increased level of awareness for linguistic diversity. The examples