{'fraction_non_alphanumeric': 0.03728572557074018, 'fraction_numerical': 0.016825973615609446, 'mean_word_length': 4.994318181818182, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
ParaShoot: A Hebrew Question Answering Dataset

NLP research in Hebrew has largely focused on morphology and syntax, where rich annotated datasets in the spirit of Universal Dependencies are available. Semantic datasets, however, are in short supply, hindering crucial advances in the development of NLP technology in Hebrew. In this work, we present ParaShoot, the first question answering dataset in modern Hebrew. The dataset follows the format and crowdsourcing methodology of SQuAD, and contains approximately 3000 annotated examples, similar to other question-answering datasets in low-resource languages. We provide the first baseline results using recently-released BERT-style models for Hebrew, showing that there is significant room for improvement on this task.


Introduction
Natural language processing has seen a surge in the pretraining paradigm in recent years with the appearance of pretrained models in a plethora of languages, including Hebrew (Chriqui and Yahav, 2021;Seker et al., 2021). While such models have shown to perform remarkably well on a variety of tasks, most of the evaluation of the Hebrew models, however, has been focused on morphology and syntax tasks in the spirit of universal dependencies (Nivre et al., 2017), while end-user-focused evaluation has been limited to sentiment analysis (Chriqui and Yahav, 2021) and named entity recognition .
In this paper, we try to remedy the scarcity of semantic datasets by presenting PARASHOOT, 1 the first question answering dataset in Hebrew, in the style of SQuAD (Rajpurkar et al., 2016). We follow similar work in constructing non-English question answering datasets (d'Hoffschmidt et al., 2020;Mozannar et al., 2019;Lim et al., 2019, inter alia), and turn to Hebrew-speaking crowdsource workers, asking them to write questions given paragraphs sampled at random from Hebrew Wikipedia. Through this process, we collect approximately 3000 annotated (paragraph, question, answer) triplets, in a setting that may be suitable for few-shot learning, simulating the amount of data a startup or academic group can quickly collect with a limited annotation budget or a short deadline.
Statistical analysis of PARASHOOT shows that the dataset is diverse in question types and complexity, and that the annotations are of decent quality. We provide baseline results based on two recentlyreleased BERT-style models in Hebrew, showing that there is much potential in devising better pretraining and fine-tuning schemes to improve the performance of Hebrew language models on this dataset. We hope that this new dataset will pave the way for practitioners and researchers to advance natural language understanding in Hebrew. 2

Dataset
We present PARASHOOT, a question answering dataset in Hebrew, in a format that closely follows that of SQuAD (Rajpurkar et al., 2016). Each example in the dataset is a triplet consisting of a paragraph, a question, and a span from the paragraph text constituting the answer to the question. We scrape paragraphs from random Hebrew Wikipedia articles, and crowdsource questions and answers for each one, resulting in 3038 annotated examples. While larger datasets may facilitate betterperforming models, recent work has advocated for research on smaller labeled datasets (Ram et al., 2021), which more accurately reflect the amount of data a startup or academic lab can collect in a short amount of time and resources. Figure 1: The annotation user interface, containing the article's title, the paragraph, a slot for entering a question, and an additional slot for entering the answer. Dragging the mouse over a span in the paragraph automatically fills the question slot, allowing for quick and accurate annotation of answer spans.

Corpus
We collect random articles from Hebrew Wikipedia, covering a wide range of domains and topics. We only sample articles containing at least two paragraphs and 500 characters. 3 Finally, for each such article, two candidate paragraphs are randomly sampled and added to the annotation corpus. These paragraphs will eventually become the passages in the question answering dataset.

Annotation
We recruit annotators by using the Prolific crowdsourcing platform. 4 Being a native Hebrew speaker is the only required qualification, allowing the participation of a few dozen annotators in the campaign. Annotators are presented with random paragraphs from the annotation set, and tasked to write 3-5 questions that are explicitly answered by the given text, for each paragraph. As in the original SQuAD annotation campaign, annotators are instructed to phrase the questions in their own words, and highlight the minimal span of characters from the paragraph that contains the answer to each ques- 3   tion. Our implementation also provides automatic data validation heuristics that alert the annotators if, for instance, the answer span is too long or not a substring of the paragraph. Figure 1 shows a screenshot from the annotation web page. 5 We acknowledge the fact that this data collection technique is known to encourage annotation artifacts (Gururangan et al., 2018;Kaushik and Lipton, 2018), and several newer annotation methods, such as TyDi QA (Clark et al., 2020), have been introduced to alleviate them. Nevertheless, we follow SQuAD's annotation methodology, as it necessitates considerably fewer resources. Maintaining an hourly wage of over $10, 6 we were able to collect our entire dataset, including discarded data from development runs, for under $800.

Post-Processing
In total, we amass 3106 question-answer examples. Of those, we discard 68 examples (2.2%) that contained yes/no questions or extremely short/long answers. The resulting dataset contains 3038 examples, which we divide to training, validation, and test by article, preventing content overlap. Table 1 details the amount of unique articles, paragraphs, and questions of each split.

Analysis
We analyze the dataset in various ways to assess its quality and limitations as a benchmark.

Annotation Quality
To measure the quality of the annotated data, we randomly select 50 examples from the validation set, and manually analyze them ourselves. 7 Specifically, we check whether the annotated answer span is correct (answers the question) and minimal (contains only the answer). Table 2    majority of the annotations are indeed valid, answering the questions with a minimal span. Yet, a significant minority contains additional supporting information, which makes the answer span longer than the desired minimal span by 2.5 times on average. We can thus expect an upper bound of 57% token F1 on those examples, setting the performance ceiling at around 84% F1 for the entire dataset. Finally, we present examples from the validation set that illustrate the annotation quality (Figure 3).

Question Diversity
To measure the dataset's diversity, we cluster questions by their question word (typically the first word in the question). Table 3 shows that what ( ‫)מה‬ and which ( ‫)איזה‬ questions account for a third of the sample, with other answer types being distributed in a rather balanced distribution. We also observe that about 11% of the data contains how ( ‫)אי‬ and why ( ‫)למה‬ questions, which may reflect more complex instances.

Sequence Length
We measure the length in words (using whitespace tokenization) of each question and each answer. Figure 2 shows the distributions of annotated questions and answers. We observe that most questions use between 4-7 words, which is typical of simple questions in Hebrew. More complicated questions constitute 27.6% of the data, for example: ‫גילברט?‬ ‫שכתבו‬ ‫האחרונה‬ ‫האופרה‬ ‫נקראת‬ ‫אי‬ ‫יחדיו‬ ‫וסאליב‬ (What is the last opera written jointly by Gilbert and Sullivan called?) There are even questions with only 2 words; due to Hebrew's rich morphology, these questions are usually translated to 3-4 words in English, e.g. ? ‫המניכאיז‬ ‫מהו‬ (What is Manichaeism?) Answer lengths, however, can vary greatly, depending on whether the annotators wrote minimal spans (typically 1-4 words) or included supporting information in the answer spans (see Section 3.1).

Linguistic Phenomena
As a morphologically-rich language (Tsarfaty et al., 2010;Seddah et al., 2013), modern Hebrew exhibits a variety of non-trivial phenomena that are uncommon in English and could be challenging for NLP models . We can identify some of these phenomena in our dataset. Consider for example the following question-answer pair from the validation set: This example illustrates a morphological variation between the question and the answer: the same entity appears as a morpheme in a compound word in the question's text: ‫שטחו‬ (its area), ‫כשהוק‬ (when it was established), but as a standalone word (i.e. without inflection) in the answer: ‫שטח‬ (area), ‫הוק‬ (was established). These phenomena make exact match-optimized predictions more difficult for models aimed to solve this task.

Baselines
We establish baseline results for PARASHOOT using BERT-style models. Results indicate the task is challenging, leaving much room for future work in Hebrew NLP to advance the state of the art.

Experiment Setup
We fine-tune three adaptations of BERT (Devlin et al., 2019): mBERT, trained by the original authors on a corpus consisting of the entire Wikipedia dumps of 100 languages; HeBERT (Chriqui and Yahav, 2021), trained on the OSCAR corpus (Ortiz Suárez et al., 2020) and Hebrew Wikipedia; AlephBERT (Seker et al., 2021), also trained on the OSCAR corpus, with an additional 71.5 million tweets in Hebrew. All models are equivalent in size to BERT-base, i.e. 12 layers, 768 model dimensions, and 110M parameters in total.
We fine-tune the models using the default implementation of HuggingFace Transformers (Wolf et al., 2020). We select the best model by validation set performance over the following hyperparameter grid: learning rate ∈ {3e−5, 5e−5, 1e−4}, batch size ∈ {16, 32, 64}, and update steps ∈ {512, 800, 1024}. We compare the models' predictions to the annotated answer using token-wise  F1 score and exact match (EM), as defined by Rajpurkar et al. (2016). Table 4 shows the performance of each model on PARASHOOT, with mBERT achieving the highest performance (56.1 F1). We also observe significant variance across the models, with mBERT and AlephBERT performing significantly better than HeBERT. It is not immediately clear where this discrepancy stems from; one possibility is that the introduction of noisy data via multilinguality (mBERT) or tweets (AlephBERT) makes that model more robust to potential noise in the annotated questions (e.g. typos). Comparing these results to the estimated ceiling performance of 84 F1 (see Section 3.1), we can infer that PARASHOOT poses a genuine challenge to future Hebrew models and encourages further analysis of the semantic capabilities of the current models.

Error Analysis
We analyze the error distribution by sampling 50 examples from the validation set and comparing AlephBERT's predictions to the annotated answers. Table 5 shows how the examples are distributed into five categories, accounting for every type of overlap between the model's prediction and the annotated answer. Putting aside exact matches (which account for about a quarter of examples), nearly half of the errors stem from zero overlap between the annotated answer and the model's prediction. We observe that a significant part of the sample (22%) contains cases where the annotated answer is a substring of the model's prediction, which might be, to a large extent, an artifact of the long answer annotations we observe in Section 3.1. For examples of erroneous predictions see Appendix A.

Conclusion
In this paper, we present PARASHOOT, the first question answering dataset in modern Hebrew, in a style and data collection methodology similar to that of SQuAD. Baseline results demonstrate the  Figure 3: Examples from the validation set. The text in bold shows crowd-annotated answers. The underlined text represents the (expert-annotated) minimal answer span. The first example demonstrates a non-minimal span that has some overlap with the question's text. The second example demonstrates a valid minimal span selection.

Overlap Type
Sample Error Frequency Frequency  Figure A.1: Predictions made by fine-tuned Aleph-BERT vs. annotated answers. In the first example, the prediction produced by the model is clearly an error. In the second example, the annotated answer span is excessively long, and the model predicts a more accurate substring of this span. In the third example, the model predicts a full sentence, while the annotated answer span is shorter.