{'fraction_non_alphanumeric': 0.03219678981859626, 'fraction_numerical': 0.012536803115205622, 'mean_word_length': 4.943832909963308, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
HerBERT Based Language Model Detects Quantifiers and Their Semantic Properties in Polish

The paper presents a tool for automatic marking up of quantifying expressions, their semantic features, and scopes. We explore the idea of using a BERT based neural model for the task (in this case HerBERT, a model trained specifically for Polish, is used). The tool is trained on a recent manually annotated Corpus of Polish Quantificational Expressions (Szymanik and Kieraś, 2022). We discuss how it performs against human annotation and present results of automatic annotation of 300 million sub-corpus of National Corpus of Polish. Our results show that language models can effectively recognise semantic category of quantification as well as identify key semantic properties of quantifiers, like monotonicity. Furthermore, the algorithm we have developed can be used for building semantically annotated quantifier corpora for other languages.


Introduction
Quantifying expressions or quantifiers are understood in this text as language expressions which indicate quantity. These can be exact numbers of objects ('three [books]') or generalised quantifiers: 'all', 'each', 'some', 'none', 'majority [of]'. Quantifiers can also count events or express their frequency: 'never', 'always', 'twice', 'often', 'repeatedly', 'each Tuesday' (Szymanik and Kieraś, 2022). The task of detecting quantifiers in the text is superficially similar to named entity recognition. The set of possible quantifiers comprises numerals expressed with digits and a seemingly limited set of other phrases, which, with some effort, could be listed in a dictionary. However, unlike in the case of named entities, not all occurrences of these expressions are in fact quantifiers. For example the digit '3' denotes a quantifier in the contexts where it is read as 'three' but not when it means 'third'. Similarly, nouns naming numbers, e.g. Polish trójka, may express a similar concept as numerals (trójka chłopców 'three boys') or it may denote any object bearing the number 3 (a radio station, TV channel, military unit, squad). In this latter context the word is not a quantifier. For another example, consider the word most. It can denote a quantifier when followed by a plural noun as in most men or a modifier when followed by an adjective as in most beautiful. Thus, the decision whether a given expression is a quantifier is to a large extent based on the semantic analysis of the linguistic context in which the expression appears. Beyond detecting quantifiers, we are also interested in two subordinate tasks: having a quantifier we want to characterise its semantic features (e.g., monotonicity) and to determine its scope, that is to find in the text the expression specifying what is being quantified. Furthermore, by showing that natural language model based on BERT can recognise quantifiers and their semantic features, we also contribute to understanding of semantic knowledge of language models, see, e.g., (Linzen et al., 2018;Linzen et al., 2019).

The Corpus and its Annotation Scheme
Although the use of quantifying expressions in natural languages has been studied in linguistics and philosophical logic for many decades now (Peters and Westerståhl, 2006;Szymanik, 2016), a corpus linguistic and computational approach to the problem is relatively new. Very few resources exist for such research, in particular large datasets suitable for machine learning methods are scarce. According to our knowledge, the only large-scale attempt at the manual annotation of generalized quantifiers was made for Polish (Szymanik and Kieraś, 2022), thus in our paper we use the Corpus of Polish Quantificational Expressions (CPQE) as the only manually annotated data set. Recently, there has been an effort to establish an ISO standard annotation scheme for quantification phenomena in natural language as part of the ISO Semantic Annotation Framework (ISO 24617) (Bunt, 2020), which could be seen as a growing interest in the field promising more development to be conducted and data sets to be published in the near future. However, the developments of the ISO standard are still very much in the preliminary stage and the first proposal of the standard was published after CPQE annotation was concluded. Technically, the corpus of Polish quantifying expressions was created by adding a new annotation layer to the gold-standard 1.2 million tokens large subcorpus of the National Corpus of Polish (Przepiórkowski et al., 2012). This corpus comprises a balanced set of short samples (approx. 40-60 words long up to full sentences) representing different text genres. The corpus of quantifiers is available on the web both as a separate layer of annotation together with the whole NKJP1M indexed in the corpus search engine as well as an XML source 'Journalists have repeatedly reported on the irregularities.' Figure 1: Quantifier annotation in CPQE. Quantifiers (in bold) are annotated with features discussed in the text. Angle brackets mark quantifier scopes. The word decision-makers is the common scope for quantifiers twenty-three and 437. The quantifier none has no scope.
tarball (Szymanik and Kieraś, 2022) 1 . The latter form, encoded according to TEI P5, was used for training neural models presented in this article. In CPQE, two main types of quantifiers are distinguished (see above for an example): • D-quantifiers are a part of dependant expressions in the predicate argument structure of sentences. Typically, they are determiners in nominal phrases.
• A-quantifiers directly build or modify predicates. Typically, these are adverbs modifying verbs.
The manually introduced annotation in the corpus consists of three elements. First, the quantifier itself is marked in the text (as a continuous sequence of words). Second, it is described using the following features: • Type: D-quantifiers and A-quantifiers (see above) • Subtype distinguishes between existential (intersective), e.g., some, universal (co-intersective), e.g., all, proportional, e.g., many, and numeral quantifiers (exact amounts), e.g., 5; see (Keenan and Paperno, 2017) for the exact definition.
• Monotonicity describes quantifier's monotonicity, left and right monotonicity is annotated as two separate features but with the same range of values: increasing, decreasing, and non-monotonic. See (Peters and Westerståhl, 2006) for the definition.
Finally, a scope is assigned to the quantifier, showing what is being quantified. By a convention, a maximal nominal phrase is marked as the scope of a D-quantifier and a full verbal form (including potential negation particles, reflexive markers, and auxiliaries) as the scope of an A-quantifier. The scope can be empty, as it may be omitted in the text. The scope may also be shared by more than one quantifier. Detailed information about considered quantifiers, their features, and rules of annotation can be found in the paper (Szymanik and Kieraś, 2022).

The Proposed Method
The process of automatic marking up quantifiers can be divided into three tasks: detecting quantifiers, classifying quantifiers, and detecting their scopes. Detection of quantifiers means deciding that some continuous spans of the text form quantifying expressions. Classification involves labels consisting of 4 parts with theoretical number of 2 × 4 × 3 × 3 = 72 combinations but in practice 58 labels are used. The quantifier scope is again a continuous span of text, but in this case a quantifier, for which the scope is sought, needs to be specified as part of the input data. In our first attempt at quantifier detection, we've tried methods successfully used for named entity recognition. Conditional Random Fields (Sutton and McCallum, 2012) are used for example in Liner2 tool (Marcińczuk and Janicki, 2012). We used a wide range of hand crafted features in our experiments: local features like base form, part-of-speech information and those taken from dependency trees created using COMBO tool (Rybak and Wróblewska, 2018). The best we've achieved for quantifier detection was about 0.8258 F1 score. If quantifier features are included, the F1 score drops to less than 0.7. The disadvantage of this method, apart from the relatively low scores, is using hand crafted features which need to be provided by external tools and can be language dependent. In the work reported in this paper, we decided to use modern models based on neural networks, in particular on the Transformer architecture and BERT. Under this methodology a pre-trained BERT model is extended with new layers specific to the task at hand. As the pre-trained model we have selected HerBERT (Mroczkowski et al., 2021), which is the best BERTtype model available for the Polish language according to the average score on KLEJ benchmark tasks 2 (Rybak et al., 2020). We have decided to use "base" variant of the model, which is a compromise between necessary computing power (in particular GPU memory) and quality. It is worth noting that the "base" version of HerBERT outperforms some other "large" models presented on the KLEJ leaderbord. The experiments were performed using Huggingface Transformers implementation in the Torch version. The detection procedure is performed on the text divided into individual sentences.

Detection of Quantifying Expressions
Quantifying expressions are specific spans of tokens in the text. Detection of quantifiers can be expressed as a token labelling task, where each token is classified as belonging or not belonging to a quantifier expression. According to the annotation rules of CPQE, a quantifier cannot be nested in another quantifier. Thus, a simple IOB markup can be used to represent the quantifiers (B marks the first token of a quantifier, I ("inside") marks non-first tokens, O is "outside"), as illustrated in Fig

Assignment of Quantifier Features
Deciding features of a given quantifier can be described as a sequence classification task. In the context of BERT based methods, this task can be solved in a rather straightforward manner with the model Bert-ForSequenceClassification: the quantifier is given as input and a label is produced as output.
As noted before, the labels used in CPQE consist of four separate features. Thus, two options arise: either the labels are predicted as atomic values or each of the features is predicted independently. In the latter case a variant of the model is needed, where the common BERT layer is extended with four independent dense layers predicting respective features. We hoped that independent prediction would perform better, since the distribution of individual features is less skewed than the distribution of whole labels. However, both approaches showed very similar results. In consequence, we have decided to predict whole labels, since this model is conceptually simpler.

Detection of Quantifier Scopes
Quantifier scope is a span of tokens in the same sentence as quantifier expression. We model detection of scopes as an instance of extractive Question Answering, where the "question" is the quantifier and the extracted answer is its scope. Since multiple quantifiers can appear in a single sentence, a particular quantifier has to be explicitly pointed at in the input data. This type of processing can be performed using Huggingface model named BertForQuestionAnswering. Typically, the input tokens for this model include the question, a special separator token, and then the text from which the answer is to be extracted. Besides being marked with a separator, the distinction between question and text is expressed with an array of numbers (token_type_ids) containing 0s for tokens belonging to the question and 1s for tokens in the text. Thus, each input position fed to BERT comprises a token id and a type id. The above representation is not perfect for scope detection. It happens sometimes that a sentence contains multiple occurrences of the same quantifier with different scopes (e.g. 3 apples and 3 oranges). In this case we need to point the model to one of the occurrences to get the right scope in result. We propose a representation where the input is only the sentence being processed and one of quantifiers in that sentence is marked with token_type_ids (Figure 3). As it turns out, although this representation is a bit different from what the model's authors assumed, BERT is able to successfully learn from such data. In this model, a dense layer, placed on top of a pretrained BERT layer, produces two numerical pointers expressing the start and end positions of the quantifier scope in the sentence. Each of the tasks described in the last three subsections is solved using a pre-trained BERT model with respective classification layer added. However, since the tasks belong to different types, it would be difficult to train a joint model. Thus, we have decided to train the models separately.

Cross-validation
To fully utilize the available data for validation, we have performed a 5-fold cross-validation on the data of The Corpus of Quantificational Expressions (as of January 2021). Tables 1 and 2 present the results. Only documents with at least one quantifier were taken to the evaluation data. Folds were created so as to contain possibly same number of quantificational expressions using the following procedure: • sort documents by number of quantifiers in the descending order, • for each document in the list, add it to the fold which is presently smallest in terms of the number of quantifiers.
The procedure resulted in three folds with 4388 quantifiers and two folds with 4387 quantifiers. Presented scores were acquired using a model trained with following hyperparameters (determined in experiments on a development set): • 2 epochs of HerBERT model fine tuning, with 32 batch size and 1.5e-5 learning rate for quantifiers prediction (IOB); • 8 epochs of HerBERT model fine tuning, with 64 batch size and 3e-5 learning rate for scopes prediction; • 4 epochs of HerBERT model fine tuning, with 64 batch size and 3e-5 learning rate for features prediction (Tags). Table 1 presents the results of quantifiers detection. Scores for "IOB" rows were counted using seqeval library 3 . "IOB only" row present score for predicting only quantifier tokens, while "IOB + Tags" row present scores acquired when Tags are counted on system predicted quantifiers not on the gold ones (which in contrary is presented in the table 2). Last row of table 1 presents scopes detection accuracy counted on the whole token spans and predictions made using gold quantifiers. Table 2 presents scores acquired for predicting complete quantifier feature set ("complete tag") and each feature separately. The predictions were made using gold quantifiers as input.
As we can see, scores are quite satisfying for all prediction subtasks.   Table 3: Inter-annotator Agreement (Cohen's Kappa) for separate features of quantifiers based on tokens annotated by two annotators (Szymanik and Kieraś, 2022)

The Model vs Humans
To get a better understanding of how the model performs against human annotators, we have taken a look at a random sample of differences. We have compared the annotation produced by the model in all folds of crossvalidation with respective gold-standard files. From the set of differences (quantifiers missing, marked differently or with different scope), a random sample of 150 was drawn. These differences were assessed by the authors with results shown in Table 4.
Annotator was right 28% System was right 50% Both wrong 7% Partially correct 15% Table 4: Who is right when the model differs form humans in annotation (on a random sample of 150 differences) The system was right in the impressive 50% of differences. In 72% of these cases (i.e. in 36% of all differences) the system has spotted a quantifier that was missing in the gold-standard annotation. In 7% of differences both versions of annotation were plainly wrong, while in 15% some parts were better but other worse than in the opposing answer, producing a mixed assesment.
Our general feeling is that the model has definitely gained a human-like competence in marking quantifiers. The differences mostly show up in places where the decision is not obvious. It is also plain that the complete list of differences is a great indication of spots in the gold-standard corpus that are potential errors or which are tagged in an inconsistent manner. An interesting question may also be whether the model simply memorises the list of quantifiers seen in the training data. The answer is clearly negative. It is most obvious with numbers written in digits: the model has clearly gained the insight that any such formation may be a quantifier. But there are also examples of more interesting quantifiers which were recognised by the system although they did not appear in the training data. For example in the following sentence the program has detected the noun mnogość 'multitude' and correctly marked a long coordinated nominal phrase as its scope: (

The NKJP Annotation
The approach presented in the paper was used to automatically annotate quantifying expressions in the large reference corpus, namely the 300 million tokens large National Corpus of Polish 4 . The quantifiers were indexed along with other layers of annotation (morphosyntax, dependency syntax, named entities) in the web-based search engine (Brouwer et al., 2017). Users may query for any instance of a quantifying expression in the corpus with respect to its orthographic form and quantifier features as well as other linguistic information contained in other layers of annotation. For example, one can query for all D-type existential quantifiers containing root element of the dependency tree 5 . Due to technical limitations of the search engine only quantifiers and their features were indexed, while the quantifiers' scopes were omitted and cannot be queried. Table 5 presents percentage of quantifier values according to the annotation scheme in the automatically annotated 300 million tokens large balanced National Corpus of Polish divided into main genres distinguished in the corpus. Each cell shows the percentage of quantifiers assigned a certain feature's value in a certain genre, e.g. in non-fiction 85.82% of recognized quantifiers are D-quantifiers and only 14.18% of them are A-quantifiers. The last two rows present a total percentege for each value in the automatically annotated balanced corpus (NKJP300) compared with the manually annotated tranining data (NKJP1M). First of all it is interesting to note that the values in those two last rows are almost the same, which shows that the distribution of the feature values are similar. As expected, one of the most large groups among the quantifiers is unmodified numerals (29.82% in training data and 30.00% in the automatically annotated corpus). Dquantifiers, including the unmodified numerals are almost ten times more frequent than A-quantifiers. Existential quantifiers (34.01% and 32.62% respectively) are more frequent than universal ones (17.79% and 16.98%), which are again slightly more frequent than proportional (18.38% and 20.41%). These frequency numbers are in line with the semantic complexity predictions for a restricted domain of 36 English quantifiers described by (Szymanik and Thorne, 2017) (see also discussion in (Szymanik and Kieraś, 2022)).

Conclusions
The main contributions of this paper are twofold. First, an algorithm has been proposed which exhibits a human-like performance in marking quantifying expressions in the text. This shows that the knowledge 4 nkjp.nlp.ipipan.waw.pl/ 5 The query for such quantifiers would look like this: <q="D:exst:. * " /> containing [deprel="root"]  embedded in language models provides enough hints for detecting such abstract notions like quantifier, scope, or monotonicity. A novel element in the algorithm is the task representation used for detecting quantifier scopes (Section 3.3.). Experiments were conducted with Polish data, but no element of the algorithm is language dependent. With appropriate data the results should scale to other languages. Second, a large corpus NKJP300 has been processed using the algorithm and the results were made available via a corpus search engine. While due to copyright reasons we are not able to release the annotated texts, we hope that corpus searches will enable interesting research on quantifiers in language. The evaluation of algorithm's performance against humans shows the necessary direction of follow-up work. The first step should be to correct gold standard data by comparing it with results of the system. We are convinced that the system already is more consistent than humans, in particular it is less prone to overlooking quantifiers in the text. Only after such cleanup it may make sense to return to fine-tuning the parame-ters of the algorithm. Also, further experiments with the model should be performed to understand how the model represents and detects the semantic properties, see, e.g., (Jumelet et al., 2021).