{'fraction_non_alphanumeric': 0.029826256586023957, 'fraction_numerical': 0.009115604091083116, 'mean_word_length': 5.104162029824171, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
SIGMORPHON–UniMorph 2022 Shared Task 0: Generalization and Typologically Diverse Morphological Inflection

The 2022 SIGMORPHON–UniMorph shared task on large scale morphological inflection generation included a wide range of typologically diverse languages: 33 languages from 11 top-level language families: Arabic (Modern Standard), Assamese, Braj, Chukchi, Eastern Armenian, Evenki, Georgian, Gothic, Gujarati, Hebrew, Hungarian, Itelmen, Karelian, Kazakh, Ket, Khalkha Mongolian, Kholosi, Korean, Lamahalot, Low German, Ludic, Magahi, Middle Low German, Old English, Old High German, Old Norse, Polish, Pomak, Slovak, Turkish, Upper Sorbian, Veps, and Xibe. We emphasize generalization along different dimensions this year by evaluating test items with unseen lemmas and unseen features separately under small and large training conditions. Across the five submitted systems and two baselines, the prediction of inflections with unseen features proved challenging, with average performance decreased substantially from last year. This was true even for languages for which the forms were in principle predictable, which suggests that further work is needed in designing systems that capture the various types of generalization required for the world’s languages.


Introduction
Generalization, the ability to extend patterns from known to unknown items, is a critical part of morphological competence. Morphological systems, both human and machine, must be able to recognize and produce novel items as new words are encountered. Every learner, every speaker, and any system intended for general use constantly encounters new words, both new coinings and existing words that are new to them.
The centrality of generalization is emphasized by the morphological sparsity that pervades language use. Inflected forms, lemmas, and inflectional categories are all sparsely distributed and highly skewed in any input sample, following longtailed, often Zipfian, frequency distributions (Chan, 2008). This has serious implications for learning, since the overwhelming majority of lemmas, if present at all in the input, will only be attested in a fraction of their possible forms. This is true even for a language like English, with only five inflected forms per verb and two per noun, and the problem only grows as a language's paradigms increase in size and complexity.
In principle, there are at least two kinds of generalization which can be evaluated in our UniMorphbased test paradigm: generalization to unseen lemmas, and generalization to unseen inflectional categories (i.e., unseen feature sets). Contrasting seen and unseen lemmas and categories yields four different test conditions: 1) prediction of the form of a novel combination of a seen lemma and seen feature set, 2) prediction given a seen lemma but novel feature set, 3) prediction given a seen feature set but novel lemma, and 4) the prediction of a form when both the lemma and feature set are novel.

Motivation for Generalization Task
Generalization to the unseen is a challenging task, the feasibility of which should be sensitive to the organization of a given language's morphology. For a language with rampant unpredictable stem mutations or suppletion, it may not always be possible to generalize patterns accurately to unseen lemmas, but one would hope that a system could generalize well for a language with invariant stems or highly irregular stem changes. Similarly, it may not be possible for a system to generalize to unseen categories for a highly fusional language where forms cannot be predicted from their component features, but it should be possible for highly agglutinative languages where roughly each feature corresponds to its own morphological operation or for a language with a high degree of syncretism in which the expression of an unseen inflectional category is  likely the same as one that has already been learned. This was shown to be feasible in practice for Nen, a Papuan language with a large degree of syncretism (Muradoglu et al., 2020). Previous iterations of this shared task have looked at some aspects of this problem, but none made this a focus. Last year's task (Pimentel et al., 2021) reported separate performance numbers for seen and unseen lemmas, but did not control for seen/unseen feature overlap. The 2018 task (Cotterell et al., 2018), sampled train and test sets with frequency weighting from Wikipedia, which made for a more naturalistic sparse sampling setting, but did not control for either kind overlap. In preparation for this year's iteration, we found that the proportion of test items with seen feature sets varied greatly across languages in the 2018 task and may have been a major driver of performance.
For example, the best performing system on Turkish, consistently scored just under the proportion of test items with seen feature sets at each training size (Table 2), even though Turkish is a agglutinative language for which generalization to unseen categories should be possible. Table 1 provides a partial noun paradigm from Turkish UniMorph which illustrates why this type of generalization should be possible. Say the feature sets N;ACC;SG and N;DAT;PL were never attested in training, but the lemma guakamole was. It should be possible to deduce their forms anyway -this would be a fair homework problem for an undergraduate course.
Looking at the table, -ler-corresponds to PL here, ∅ to SG, and -in-to PSS3S. Both forms with ACC end in -i, while DAT seems to correspond to -ye in the singular and -e in the plural. From this alone, one can correctly infer that N;DAT;PL should be guakamole-ler-e, while N;ACC;SG should be guakamole-yi or maybe guakamole-i. The former is indeed correct: y-insertion is well attested elsewhere in the language and would certainly be present with other lemmas and with other feature sets containing ACC. While unseen Turkish inflectional categories are not completely predictable, since they also contain some morphological eccentricities which obscure predictability, "could an undergraduate solve it?" is a good rule of thumb for whether generalization to unseen feature sets is a feasible task.
Performance was divergent on closely related languages whose test sets' feature set overlaps differed. Turkish and Azeri are closely related Oghuz Turkic languages with some mutual intelligibility (Salehi and Neysani, 2017) and very similar morphological paradigms, nevertheless, scores for Azeri during the 2018 task were much higher than for Turkish. Table 3 shows feature overlap and performance for Azeri. It is tempting to propose that Azeri scores were higher than Turkish scores because overlap proportions were higher.
Taken together, this suggests two things. First, the proportion of test items with feature sets attested in training is an uncontrolled factor in the data that could be driving performance in a way that obscures language-internal factors. Second, this could suggest that the systems of the day were not able to generalize across inflectional categories, 2 but a more focused evaluation would be needed to investigate these hypotheses. We perform such an investigation this year.   Table 3: Comparison of best 2018 system accuracy on Azeri low-, medium-, and high-train conditions and percent of test items with feature sets attested during training.
2 Recent work has shown that lemma overlap is also an important predictor of performance (Goldman et al., 2022), but an analysis of 2018 results suggests that feature set overlap is an even better predictor (see Appendix A).

Task Description
From the participants' perspective, this task was organized very similarly to previous iterations. Participants were asked to design supervised learning systems which could predict an inflected form given a lemma and a morphological feature set corresponding to an inflectional category or cell in a morphological paradigm. They were provided with a small, and data permitting, large training set, as well as a development set and test set for each language. The train and dev sets consisted of (lemma, inflected, feature set) triples, while the inflected forms were held out from the test set.
Data was made available to participants in two phases. In the first phase, train and dev sets were provided, with the expectation that model development and tuning be carried out primarily on these languages. In the second phase, test sets were released for all languages during the evaluation phase. Teams produced predicted inflected forms for each test set. They were given the opportunity to submit two sets of predictions from two separate models, one trained on the small training sets and one trained on the large training sets, with the latter being a super set of the former.

Description of Languages
This section provides brief descriptions of each language that was newly included or newly updated for this year's task. Further information about returning languages can be found in previous years ' papers (Vylomova et al., 2020;Pimentel et al., 2021). Table 4 summarizes the list of languages and provides citation and attribution information.

Armenian (Indo-European)
Armenian is an independent branch of the Indo-European family. Its oldest attested form is Old Armenian or Classical Armenian (∼5 th century). It has two modern standardized varieties: Western Armenian and Eastern Armenian. Western Armenian is a diasporic language that developed in the Ottoman Empire, while Eastern Armenian is the official language of the Republic of Armenia (Dum-Tragut, 2009

Finno-Ugric (Uralic)
Finno-Ugric is a branch of Uralic, a language family with around 25 million native speakers spread between Northern Russia, Scandinavia, and Hungary. The majority of them are agglutinating and extensively use suffixes. They are also known for a relatively rich grammatical case system. Verbs are inflected for number, person, tense, and mood. Phonologically, these languages often present vowel harmony and palatalization. Hungarian, with its 13 million native speakers, is the most widely spoken Uralic language. Hungarian is an agglutinative language with a rich set of affixes expressing derivation or inflection, such as in the verb. Another feature of Hungarian morphology, adding to its complexity from a computational perspective, is vowel harmony: the vowels of certain affixes adapt to those of the stem (Rounds, 2009). Compounding in Hungarian is frequent and productive, leading to further complexity in its morphological analysis (Kiefer and Nemeth, 2019).
Karelian and Ludic are two closely related Finnic varieties spoken in Russian and Finnish Karelia and the regions around Lakes Onega and Ladoga. The data for both languages, along with Veps returning from last year, has been collected as part of the VepKar project (Boyko et al., 2021) and includes multiple dialects. Typical of Finnic, these languages are highly agglutinative, present vowel harmony processes, and overtly express well over ten cases on nominals and adjectives. Ludic is often described as a dialect of Karelian, although it has certain unique features such as the presence of a reflexive conjugation (Novak et al., 2019) and the use of the full temporal paradigm of the conditional. It is seriously endangered, with about 150 remaining speakers. 3

Georgian (Kartvelian)
Kartvelian, or South Caucasian, languages are primarily spoken in the South Caucasus with no demonstrable genetic relation to other languages in the region. Georgian, an official language of Georgia, has about four million speakers worldwide. Georgian morphology is mostly agglutinative. Nouns have number (singular/plural), but no grammatical gender. Its grammatical case system is relatively rich, having seven cases. Nouns are declined for number and case. Verbs exhibit polyper-sonal agreement (incorporating the number and the person of both subject and objects). In addition, verbs are divided into 4 classes: transitive, intransitive, indirect, and medial, and present many irregularities.

Germanic (Indo-European)
The Germanic family constitutes one of the primary branches of Indo-European. It in turn contains three sub-branches. The West Germanic sub-branch includes English, Dutch, and German, among others. The North Germanic sub-branch contains the Germanic languages of Scandinavia. The East Germanic sub-branch is extinct and contained Gothic. At a high level, Germanic morphology is similar to that of other Indo-European branches, but it does diverge in some key ways (Ringe, 2017). Germanic languages, particularly in the past, had an inherited three-way gender distinction, an inherited three-way number system, and overt inflectional case systems, all reduced to some degree from Indo-European. Nominals fall into several inflectional classes with different case/number expressions. 4 The 2020 shared task revealed some major inconsistencies in the data (Vylomova et al., 2020). In this iteration, the data has been re-extracted and checked.
Gothic is an extinct East Germanic language. Nearly the entire extant Gothic corpus comes from a partial translation of the Christian Bible by bishop Wulfila. Gothic is in many ways more conservative than other Germanic languages. It lacks Umlaut, which is a type of vowel alternation on nouns and verbs present in the rest of the family, but it retains reduplicated perfects, and it sometimes uses the accusative as a vocative case. Data for Gothic was sourced from Wiktionary and contains both Gothic script and Latin transcriptions.
Old English, Old High German, and Old Norse were three closely related West and North Germanic languages and early attested ancestors of modern English, High German varieties, and liv-4 Five of the six Germanic languages presented this year are historical. They no longer have living speakers, and their corpora are of a fixed size. Paradigms were initially extracted from Wiktionary. Given the highly skewed long-tailed distributions of inflected forms, lemmas, and inflected categories (Chan, 2008), which do not differ in historical corpora (Kodner, 2019), the large majority of potential inflected forms, even for known lemmas, are not attested in the historical record. As such, most of the forms in the full paradigms available on Wiktionary are generated and not actually attested. This is likely not a major concern for the purpose of this task, but the caveat must be expressed.

180
ing North Germanic languages today. Inflectional classes in these languages are often less transparent than in Gothic due to successive sound changes obscuring their basis.
Middle Low German was a collection of West Germanic dialects spoken along the southern North Sea coast. It was a major trade language, the lingua franca of the Hanseatic League during the European Medieval period. The language retains overt case distinctions on nominals, but it shows a greater degree of syncretism than earlier Germanic languages. This trend of increased syncretism extends to the verbal system as well (Lasch, 1914).
Low German is a collection of West Germanic varieties descended from Middle Low German occupying an intermediate space in a dialect continuum between Dutch and High German. Varieties exist in a state of diglossia, mostly with Standard German, a High German variety. Several million native speakers remain in the 21st century, though numbers are declining. Outside of Europe, Low German is spoken in some diaspora communities including Mennonite groups in the Americas.

Hebrew (Semitic)
The Semitic languages, a branch of the larger Afro-Asiatic family, are spoken by over 300 million people across North Africa and Southwest Asia. Hebrew is a Northwest Semitic language with around 5 million native speakers, spoken mainly in Israel. Typically of Semitic, Hebrew makes heavy use of templatic non-concatenative morphology (Coffin and Bolozky, 2005). Verbs are expressed through triliteral consonant roots which occupy slots in a template of vowels. Verbs occupy inflectional classes called binyanim in Hebrew. Person, number, and tense marking is indicated primarily with affixation. Both prefixation and suffixation are applied depending on the tense. Nouns and adjectives indicate gender and number through suffixation, sometimes with stem mutations. Verbs, nouns, and propositions may take possessive or pronominal object clitics. In the current shared task we introduce a vocalized version of Hebrew that has been recently added to the UniMorph.

Indic, or Indo-Aryan (Indo-European)
Indic is a branch of Indo-Iranian, itself a primary branch of Indo-European. The family has a long history, with a large attested corpus of Vedic and Classical Sanskrit. It currently has over 800 million speakers extending through all countries in South Asia. Morphologically conservative languages express a three-way gender distinction and case on nouns, tense, aspect, mood, number, and person on verbs. Inflectional morphology is primarily suffixing. Some languages possess overt formality distinctions on verbs.
Assamese is mainly spoken in the northeast Indian state of Assam, with over 20 million native speakers. While gender is not grammatically marked, Assamese presents a rich system of noun classifiers. The Assamese data has been extracted from the English edition of Wiktionary. Gujarati (Baxi et al., 2021) is spoken predominantly in the Indian state of Gujarat, with over 50 million native speakers. Kholosi is an under-documented Indo-Aryan language spoken in two villages (Kholus and Gotav) in Hormozgan Province, Iran. The data has been collected during field work (Arora and Etebari, 2021).

Ket (Yeniseian)
Yeniseian languages were historically spoken along the Yenisei River region of central Siberia. Ket, the only living member, is critically endangered, with only about 60 remaining speakers at any level of linguistic competence. The language presents mainly agglutinative morphology, with extensive use of suffixes, prefixes, and infixes. Although verbal conjugation and noun declension systems are well-developed, the boundaries between word classes are fuzzy (Verner, 1997). Noun classes differentiate between masculine and non-masculine in the singular, animate and inanimate in the plural. The grammatical case system contains between 8 and 10 cases depending on the analysis. Ket verbs express polypersonal agreement, with the case and number of all arguments reflected on the verb.
The data for Ket was sourced from a text collection compiled during the field work of the Laboratory for Computational Lexicography of the Moscow State University, that took place between 2004 and 2009. It contains word forms from twelve categories, seven of which (ADJ, NUM, ADV, INTJ, ADP, PART, CONJ) are invariable.

Khalkha Mongolian (Mongolic)
The Mongolic language family has 5.200 million active speakers of 14 language varieties, which are actively spoken in Mongolia, Russia, China, and Afghanistan. The Khalkha Mongolian is de facto the official national language of Mongolia and both the most widely spoken and most-known member of the Mongolic language family. Khalkha Mongolian is an agglutinative language with a rich set of suffixes, but no prefixes. It also expresses complex vowel harmony patterns (Jaimai et al., 2005).

Korean (Koreanic)
Korean, spoken by about 80 million people, is often described as a language isolate. However, the Jeju dialect, spoken on the southern island of Jeju is highly divergent and often considered its own language. The language expresses limited inflectional morphology on nominals. Verbs express valency, tense, aspect, mood, and various dimensions of formality through suffixation. The current dataset consists of mostly predicates, so the resulting lemmas are mainly verbs and a smaller number of adjectives.

Lamahalot (Austronesian)
Lamahalot, or Solor, is one of the Central-Malayo-Polynesian languages, a proposed branch in the Malayo-Polynesian within Austronesian. As of 2010, it had about 200,000 native speakers, primarily on the eastern part of Flores Island, and neighboring islands of Flores (Solor, Adonara, Lembata, and Alor). Nearby Papuan languages have had a significant influence on this language phonologically and syntactically (Nagaya, 2011;Arka, 2007;Klamer, 2002Klamer, , 2009. The language has several dialects. We use data mainly from the Lewotobi dialect (Nagaya, 2011) spoken by about 6,000 people in Kecamatan Ile Bura, East Flores. Morphologically, Lamaholot is a nearly isolating language (each word typically has one morpheme) with a small inventory of affixes (mostly prefixes and a handful of suffixes) and clitics (mainly enclitics). This language has two salient morphological features, namely agreement and nominalization.

Slavic (Indo-European)
Slavic, another primary branch of Indo-European, contains approximately 20 languages, with half of them having over 1 million speakers. The languages are spoken in Central and Eastern Europe, the Balkans, and Russia. They are traditionally divided into three branches: East Slavic (incl. Belarusian, Russian, Rusyn, and Ukrainian), West Slavic (incl. Czech, Kashubian, Polish, Silesian, Slovak, and Upper and Lower Sorbian, among others), and South Slavic (incl. varieties of Bosnian-Croatian-Montenegrin-Serbian, varieties of Macedonian and Bulgarian including Pomak, and Slovenian).
Slavic morphology is generally typical of Indo-European, with several inflectional classes for both verbs and nouns, nominal inflection by case, number, and three genders. It elaborates Indo-European verbal inflectional paradigms marking aspect, tense, number, person, and sometimes gender.
Slovak (Mistrík, 1988), and Upper Sorbian are two closely related West Slavic languages. Masculine nouns additionally mark animacy, which is often described as a part of the gender system of these languages. The case systems of both languages are fairly similar, however in Slovak, vocative is usually syncretic with nominative. Upper Sorbian retains a dual number and has a greater variety of verbal past forms than other West Slavic languages. The Slovak data was obtained by automatic conversion of extensive inflectional dictionaries used for morphological analysis to the UniMorph scheme. 5 The data for Upper Sorbian was combined from WMT and online grammars. 6 Pomak is a South Slavic language, a dialect of Southeastern Bulgarian spoken in Greece and European Turkey. It has around 30,000 speakers as of 2021 but lacks standardized orthography (Jusúf Karahóǧa et al., 2022). Bulgarian and Macedonian varieties are unusual among Slavic for having mostly lost case marking on nouns and for marking voice synthetically on verbs.

Data Preparation
All data for this task is provided in standard UniMorph format, with training items consisting of (lemma, inflected form, morphosyntactic features) triples. Since the goal of the task is to predict inflected forms, the test set was presented as (lemma, features) pairs. Data was canonicalized as in previous years using https://github.com/ unimorph/um-canonicalize, which ensures consistent ordering of the features in the feature sets.

Training-Test Overlap
As always, we ensured that there are no lemmafeature set pairs that occur in both the training and test sets. However, since test items contain both lemmas and features, other overlaps between training and test are possible. This year's data splitting algorithm aimed to control for the four logically possible licit types of lemma and feature overlap, which define four kinds of test items: Both Overlap: Both the lemma and feature set of a training pair are attested in the training set (but not together in the same triple) Lemma Overlap: A test pair's lemma is attested in training, but its feature set is novel Feature Overlap: A test pair's feature set is attested in training, but its lemma is novel Neither Overlap: A test pair is entirely unattested in training. Both its lemma and features are novel.
For illustration, consider the sample training and test sets provided in (1)-(2). In this example, each test pair exhibits a different kind of overlap.

Data Splits
The data set for each language was split into training, development, and test sets. For languages with sufficiently large corpora, both large and small training sets were produced with the small set being a subset of the large one. We aimed for 7,000/1,000/2,000-item large train/dev/test splits and a 700-item small train split when possible, but splits for most languages were somewhat smaller in practice. Chukchi, Kholosi, Lamahalot, and Xibe in particular were too small to extract even full small training sets, while Braj, Gujarati, Itelmen, Ket, Low German, Magahi, Middle Low German, Old High German, Upper Sorbian were too small to extract large training sets. Split sizes are summarized in Table 5. 7

Motivation for Data Splitting
The sampling script attempts to control the size of each overlap category in the test set. The challenge here is controlling for both lemma overlap and feature overlap simultaneously. Since no frequency information is provided in the UniMorph annotation scheme, any uniform sampling over triples, controlling for lemma overlap or otherwise, will tend to drive feature overlap to near 100%. This is unnatural. Since both lemmas and inflectional categories tend to follow long-tailed sparse frequency distributions in real language (Chan, 2008, ch. 3), a naturalistic split weighted by token frequencies of individual items will tend to oversample high frequency lemmas and inflectional categories (i.e., feature sets), and undersample most others. This skewed sampling should yield a mix of overlap types in the test set. This is what was achieved in 2018, though the ratios of overlap types were uncontrolled. In contrast, this year's data splitting achieves a controlled mixture of overlap types even in the absence of frequency information.

Splitting Process
The algorithm began by randomly partitioning a language's feature sets into OVERLAPPABLE and NON-OVERLAPPABLE sets and uniformly sampling the large training set from only those triples that contain feature sets in OVERLAPPABLE. If there were not enough triples with with feature sets in OVERLAPPABLE for a given language, then the OVERLAPPABLE partition was increased incrementally until enough training triples could be sampled. If there was insufficient data to create the large training set, then the small training set was sampled this way instead. If there was enough data, then the small training set was down-sampled uniformly from the large training set.
The test set was sampled from the remaining items, with half drawn from triples with feature sets in OVERLAPPABLE and half from triples with feature sets in NON-OVERLAPPABLE features. The development set was drawn from the remainder in the same fashion.
As summarized in Table 5, this approach resulted in a much more even mixtures of overlapping pairs at both training sizes than is achieved by sampling that does not take feature overlap into account, though the actual ratios varied by language due to corpus-specific and language-specific factors. In controlling for feature overlap, a good mixture of lemma overlap items is achieved simultaneously. Since most languages provide ample attestation of each overlap type, we could evaluate on each overlap type individually to gauge models' generalization abilities across both the lemma and inflectional category dimensions. Additionally, in aiming for a more uniform ratio of overlap types across languages, overall performance on each language is more directly comparable.

Baseline Systems
The organizers provided one neural and one nonneural baseline system. The neural system, Neural, is a character-level transformer (Wu et al., 2021). It is identical to the system CHR-TRM which was used in the 2021 task. The non-neural system, NonNeur, is identical to the non-neural baseline made available in 2020 and 2021. 8

CLUZH (Silvan Wehrli and Makarov, 2022):
The CLUZH team adapted their earlier model, character-level neural transducer, to work on large datasets (Makarov and Clematide, 2020). The model has previously shown superior performance, especially in low-resource scenarios. This year, the team optimized the training procedure using minibatches. They only relied on the teacher-forcing approach, i.e., using gold labels rather than what was predicted during the training phase. Morphosyntactic features were treated individually, and their embeddings were summed. The team explored performance of the model across various task settings and demonstrated its ability to capture feature behaviour better than other team's models, especially in the small training condition. The system is identical to the one submitted to this year's acquisitioninspired subtask (Kodner and Khalifa, 2022). OSU (Elsner and Court, 2022): OSU's system is identical to the one submitted to this year's acquisition-inspired subtask. This inflection system is a transformer whose input is augmented with an analogical exemplar model showing how to inflect a different word into the target cell. In addition, alignment-based heuristic features indicate how well the exemplar is likely to match the output. The system works only when examples of the target cell are present in the training set and can serve as exemplars; otherwise, it outputs the 8 Available here: https://github.com/sigmorphon/ 2022InflectionST/tree/main/baselines/nonneural lemma as a placeholder. Thus, the system's scores are expected to be higher for the feature overlap and both overlap evaluation categories and very low when the target cell is unknown. (Merzhevich et al., 2022): Tü-Morph's neural system is a modification of the character-level adaptation of transformer to morphology from Wu et al. (2021). In particular, the team trained the transformer to predict a distribution over states of FST (whose states are characters) rather than character sequences themselves. The model is scored third on both the small and large training settings. (Merzhevich et al., 2022): As their second submission, the team manually developed FSTs using grammars and corresponding UnMorph repositories. Since that requires more human labour and linguistic competence, the team focused only on three languages: Chukchi, Kholosi, and Upper Sorbian. The resulting FST models outperformed all other submitted systems on two of three languages. The authors confirm earlier observations from Beemer et al. (2020) that such systems are able to reach superior results compared to neural ones, especially in low-resource scenarios and high morphological complexity, but require substantially more human working hours.

TüMorph-FST
UBC (Yang et al., 2022): The UBC team proposed enriching the character-level transformer of Wu et al. (2021) with reverse positional embeddings to better account for suffixing, one of the most common word formation processes. In addition, the team explored a synthetic data augmentation technique proposed by Anastasopoulos and Neubig (2019) and student-forcing (Nicolai and Silfverberg, 2020), a training strategy where the model outputs are replaced with gold labels for some percentage of samples to alleviate exposure bias. Data augmentation leads to significant improvements, especially in the small training condition, confirming its utility. The student forcing training also provides a certain accuracy gain but presents mixed results when used together with data hallucination.

Results and Evaluation
Performance was evaluated by exact match accuracy. Macro-averages across languages on the entire test set and partitioned over the four overlap types are provided in Table 6. Results by language for both small and large training conditions are provided in Tables 14-18 in Appendix B.
A few points stand out immediately. First, overall performance is much lower this year compared to last year's similar task. During the 2021 iteration, all systems achieved over 90% accuracy on most of languages, while this year, no system achieves over 72% average in either training condition. This task was designed to be particularly challenging because the test set required systems to make predictions with only partial information. The results bore out this expectation.
Flexica, the only general non-neural submitted system, surpasses the non-neural baseline, but does not surpass 40% overall accuracy in either training condition. Being a hand-built system, TüMorph-FST outperformed all other systems on two of three languages that it was developed for.
As expected, all systems that submitted full or nearly full predictions for both the small and large training conditions performed substantially better with more training data. CLUZH, TüMorph-Main, UBC, and the neural baseline each improved by over ten points, while Flexica and the non-neural baseline showed smaller gains of around four points.
UBC achieved the highest performance of any system in either training condition. To understand why this is, it is necessary to look at a breakdown of performance by overlap type. The system is more resilient to novel feature sets than any other except for the hand-built FSTs.  Table 6: Macro-average accuracy for each system. Three systems (OSU, TüMorph-Main, and TüMorph-FST) only submitted predictions for a subset of languages in the small training condition, so their numbers (italicized) are not directly comparable to the others. Flexica and NonNeur are non-neural.

Analysis by Overlap Partition
A breakdown by overlap partition reveals some consistent trends. As expected, neither overlap items proved challenging, since systems had to infer the forms for simultaneously novel lemmas and novel feature sets. Surprisingly, all systems performed better on neither overlap items than lemma overlap items. It is not clear why this would be, since it is observed on average for many but not all of the tested systems. It may be an artifact of the data splitting algorithm favoring balancing feature overlap over lemma overlap. However, the results are consistent with the observation over the 2018 data that systems struggle generalizing across feature sets more so than generalizing over lemmas. They perform better on generalizations across lemmas to such an extent that the proportion of items with feature overlap in the test set washes out the effect of seen and unseen lemmas. Tables 7-8 illustrate this point quantitatively. Table 7 compares average performance on test items with feature sets attested in training (both overlap ∪ feature overlap items) with test items with novel feature sets (neither overlap ∪ lemma overlap items). All systems perform better on items with attested feature sets, but the gap in performance varies greatly from UBC's 32 points in the small training condition to OSU's 79 points in the large training condition. OSU's drop in performance is expected because it outputs the lemma when the feature set is unknown. In these cases it makes correct predictions exactly when the inflected form is identical to the lemma, pointing to a degree of syncretism in the data. Table 8 shows the same, but for test items with lemmas attested during training both overlap ∪ lemma overlap items) and test items with novel feature sets (neither overlap ∪ feature overlap items). Every system actually performs worse on the attested lemma items than the novel lemma items.   Tables 6-7 together elucidate a clear difference between CLUZH and UBC. While the former outperforms the latter on items with seen feature sets, the latter outperforms the former on itsems with novel feature sets. This means that UBC outperformed CLUZH on this data set because it is better suited for generalization to unseen features, something that would likely been hidden if tested on previous years' data.
However, there is a sense in which testing on items with novel feature sets is not entirely fair for all languages. In highly fusional languages in particular, it may not actually be possible to predict the mapping from a set of semantic features to a particular inflection given what is known about the member features. On the other hand, it should be solvable for a canonically agglutinative language where each member feature contributes one piece of the inflected form like "beads on a string." Thus, it could be possible that the lower aggregate performance observed on novel feature test items is not due to a failure of generalization in the systems but rather the impossible nature of the task. Table 9 tests this hypothesis. It shows average performance only on languages considered to be primarily agglutinative: Chukchi, Evenki, Georgian, Hungarian, Itelmen, Karelian, Kazakh, Ket, Korean, Ludic, Mongolian, Turkish, Veps, and Xibe. Further information can be gleaned from performance on each language individually as reported in Tables 14-18 in Appendix B.
In principle, a system should be able to infer the appropriate morphological operations for unseen feature sets in these languages, as was illustrated for Turkish in Table 1. While this is not a perfect test, since real agglutinative languages also contain some morphological eccentricities which obscure predictability, "could an undergraduate solve it?" does apply. It provides a clear result: the gap between performance on test items attested and novel features does not generally improve even for these languages where it should, if the unfairness of the task were driving decreased performance on fusional languages. This shows that generalization to novel feature sets, that is, to previously unattested inflectional categories, remains a legitimate concern for nearly all the systems.

Results by Part-of-Speech
As in previous years, the data employed for this task contains items from several parts-of-speech. Languages vary considerably in how much inflection they apply to different POS categories. As such, collapsing over POS categories can obscure interesting patterns. Tables 19-26 provide results for test items tagged with the four most common part-of-speech features in this year's data: verb (V), noun (N), adjective (ADJ), and participle (V.PTCP). Given the overall challenging nature of this year's task, performance across POS categories is generally weaker than what was reported for last year.

Error Analysis by Language
This section contains qualitative error analysis for six languages from five different top-level families.

Arabic
As shown in Table 17, none of the systems outperformed either of the baselines in the overall partition in the large training setting. 15% of the lemmas in the test set were not inflected correctly by all the systems. Nouns (N) made up the majority of those errors (47.8%). Focusing on the noun majority, errors included inaccurate plurals, minor orthographic errors, and "reasonable" confusion of different state and possession features. The plural inflection errors follow a similar pattern to those in this year's acquisitioninspired subtask. See Kodner and Khalifa (2022) for more in-depth analysis. Orthographic errors include minor common mistakes resulting from missing orthotactic operations or an alternative spelling in the gold form. Lastly, there seems to be some confusion between SPEC, DEF, PSSD tags 9 in the dual and masculine plural forms since both those suffixes inflect for case and state. This confusion is mainly due to the existence of possible different forms of the same lemma sharing the same feature set or vice versa in the training data.
On the other hand, all systems correctly inflected 29% of the lemmas. In this case, 55% of those cases are adjectives (ADJ). This is not very surprising since adjectives in Arabic are more regular than nouns in pluralization in particular. Most of the plurals in this set are those ending with the feminine plural suffix, which does not inflect for case and state the same way the masculine plural suffix does. On the other hand, most of the masculine adjectives are singular and therefore the case and state inflections are easier.
In the small training setting, systems follow a similar trend, shown in Table 14. However, there is a higher percentage of verbs (V) among the lemmas that all systems inflected incorrectly. This is expected since verbal paradigms in Modern Standard Arabic tend to be very large in size, therefore, more sparsity in smaller training sets.

Armenian
Armenian orthography is quite close to the pronunciation of words. But all four models had issues when the triggers for inflectional allomorphy were from phonology, semantics, or morphological classes. 10 The different learning models had problems in respecting the rather close correspondence between the orthography and phonology. For example, given a word with a final orthographic <a> like <anjnya> 'personable', adding a vowel-initial suffix sequence like -i-s (-GEN-POSS2SG) triggers a glide in both the orthography and pronunciation: <anjnyayis>. All four models incorrectly generated a glideless form for this word <*anjnyais>.
There were also cases of transparent phonological-conditioned allomorphy that caused errors. The definite suffix is <-n> after vowels, but <-@> after consonants. Given a vowel-final word like <moṙeni> 'raspberry,' the definite form should thus be <moṙenin>, yet all four models made some type of error. The Flexica model used an entirely different ablative suffix -ic', while the other three models used the wrong definite allomorph -@. This allomorphy rule is exceptionless and is fully transparent from the reformed Armenian orthography. These errors suggest that the models didn't fully exploit the phonological properties that are reflected in the orthography. It is possible that such errors would reduce if the models incorporated some level of 10 Transliteration is the Hübschmann-Meillet-Benveniste (HMB) system: https://en.wiktionary.org/wiki/ Wiktionary:Armenian_transliteration.
Forms in <angled brackets> are transliterations.
phonological information, such as by making the input forms be transcribed forms, and by having the models have a priori knowledge of cross-linguistic phonological feature systems.
Some errors were unavoidable and are due to phonology-semantics interactions. The plural suffix is <-er> after monosyllabic words, but <-ner> after polysyllabic words. For example, the monosyllabic word <nyut'> 'material' takes the plural <nyut'-er>. But if a word is an endocentric compound, then the plural suffix must count the number of syllables in the second stem of the word (the head). For example, the word <šparanyut'> 'makeup' is an endocentric compound of <špar> 'makeup' and <nyut'>. Its plural unambiguously takes -er because of the transparent semantic connections between the compound and the monosyllabic second stem. But all four models incorrectly generated the polysyllabic-selecting suffix -ner. It is not surprising that all four models made errors of this type. To avoid such errors, the models would need access to semantic information of the compound, and to also access the semantics of other words in the lexicon (the stems). Some errors were due to purely morphological under-learning. Armenian has many different declension and conjugation classes. The different models made over-regularization mistakes, whereby they used regular inflectional suffixes over irregular ones. Sometimes the use of a suffix triggers morphological alternations in the stem. The models however preferred to keep the shape of the stem constant. Such 'mistakes' are common in colloquial speech, but they are absent in the prescriptive declension patterns that the Wiktionary data uses.

Hungarian
The richness of the Hungarian inflection system made prediction hard for all systems. While most errors show failures of generalization, many are attributable to genuinely hard, i.e., irregular or weakly systematic, forms of inflection. Mistakes due to vowel harmony are very frequent, as the vowels to be used in inflections are often unpredictable and can only be judged in terms of frequency in everyday use. Thus, *megtilt+enélek is clearly ungrammatical (it should be megtilt+análak), but forms such as szellős+ök or objektív+től, not present in the gold standard, are actually used. Another recurrent mistake is the presence or ab-sence of the -j-in possessives where, again, systematicity is weak: in siketfajd+(j)a, the form without the -j-is not acceptable, but in other cases (hangár+(j)aitok, tranzisztor+(j)a) native speakers may accept either form. Unsurprisingly, all systems tended to fail over irregular inflections, such as hard-to-predict (but frequently used) inflectional classes, such as low vowel nouns (singular út but plural utak) or v-stems (singular ló but plural lovak). Finally, homonymy can also explain apparent mistakes, such as szél that means both wind and edge: in the first case its plural is szelek while in the second case it is szélek.

Khalkha Mongolian
Mongolian inflectional suffixes are highly unambiguous given a lemma's POS feature. Every inflectional suffix often belongs to only one morphological feature (Denwood, 2011;Munkhjargal et al., 2016). For example, Mongolian -iin belongs only to the genitive case while German -s suffix has two meanings by making the inflectional forms of either the genitive case or plural nouns. In this sense of low ambiguity, it is not surprising to see that the all participating systems have zero accuracy over the lemma overlap settings in Tables 15 and 18.

Polish
Performance on Polish was decent overall. In the small training condition, CLUZH managed to achieve nearly 91% on the lemma overlap items. While number decreased to 84% in the large training condition, which likely suggests that the lemma overlap test partitions contained coincidentally easy items, it does demonstrate generalization. Not all systems succeeded on the lemma overlap items. OSU, Flexica, and the non-neural baseline showed the usual performance drop.
Masculine genitive singular inflection proved challenging. There are two possible endings, -u and -a, but their distribution is unpredictable. As a classic example of paradigmatic gaps, native speakers themselves frequently disagree on which ending to apply (Dąbrowska, 2001). Then it is unsurprisingly that systems sometimes predict the wrong ending. For example CLUZH produced *przystępa for przystępu as the genitive singular of przystęp. It also produced filungu instead of filunga as the genitive singular of filung, which is a known variant form in the language, but not the one presented in the gold standard data.
Systems also confuse masculine and feminine forms or inflect the wrong case. They also misapply yers, or palatalization, a pervasive process in Polish and in Slavic more generally. These types of errors were also identified in an error analysis of the 2017 task in Gorman et al. (2019). See that paper for more information.

Turkish
Turkish exhibits both front/back and rounding harmony. Harmony mismatches are a major source of errors on the language. For example, Flexica produces *dokumalisin, a front/back violation for expected dokumalısın, and CLUZH produces a rounding violation *yoldurtmışım for yoldurtmuşum. Flexica, the only non-neural submitted system particularly strugged in this area.
Voicing assimilation, which can occur intervocalically and at some morpheme boundaries, also proved to be challenging. For example, Flexica and CLUZH, the stem çıldırt-ends in voiceless stop, therefore the consonant of the following past tense suffix should be devoiced and realized as [t], however, in these three systems it remains [d], thus resulting in forms like *çıldırtdım mı for expected çıldırttım mı. CLUZH and Flexica do not perform intervocalic voicing for akrebinizi from akrep and instead produce *akrepinizi. Similarly no system except for TüMorph-Main correctly produces asidi from asit. They instead produce *asiti. Related to this, systems sometimes fail to insert epenthetic glides between vowels in hiatus.
Sometimes systems produce commission errors, substituting a morpheme with one absent in the feature set. For example, for CLUZH in the small training condition, the case marking is wrong for the lemma balta: instead of producing the genitive -ın, it adds the ablative -dan even though the GEN feature is present. The same issue holds in quite a few lines as well. For example, for Flexica, the features contain GEN, but the system generates it with dative case (along with a vowel harmony error as in Hungarian), thus producing *havai fişeklara instead of the expected form havai fişeklerin. All systems struggle significantly on items with unseen feature sets. This is interesting, because Turkish should have been one of the languages most conducive to generalization over unseen feature sets. The systems may not be associating the features in a set with their corresponding agglutinative realizations.

Discussion
This year's shared task investigated two dimensions of generalization in morphological inflection: generalization over lemmas and generalization over inflectional categories. Test items with lemmas or feature sets that were attested in training were evaluated separately from those with novel lemmas or feature sets to gain a better understanding of generalization. This proved to be a challenging version of the task, as performance is substantially lower across systems compared to previous years.
We carried on the tradition of including a range of typologically diverse languages in the task. From the perspective of the two dimensions of generalization, different morphological paradigms could prove more or less challenging. In particular, it is more reasonable to expect a system to generalize to an unseen feature set if the form of the corresponding inflectional category is in some way derived from forms associated with each of the member features. Similarly, a language with relatively invariant stem forms and little unpredictable stem-conditioned realization of inflectional categories should be conducive to generalization across lemmas, while a language with more stem changes or lexically arbitrary inflectional classes should prove more challenging.
Two major patterns emerged which held across systems. First, overall averages were lower than previous years in which overlaps between lemmas and features in training and test were left uncontrolled. The task was challenging. Second, performance test items with novel feature sets was almost uniformly weaker than performance on test items with novel lemmas. This was true for all systems and still held true for agglutinative languages which stood the best chance of generalization across feature sets.

Implications for Future Work
The results of this year's shared tasks have some implications for future systems and future shared tasks. First, since overlap type has a major effect on performance, cross-linguistic differences in performance in morphological inflection tasks may sometimes be driven by these distributions rather language-internal. Since these overlaps were hardly evaluated in previous years, a reanalysis of prior years' shared tasks along these lines may uncover interesting results. Related to this, train/test/dev splits created by uniform sampling of UniMorph will not only lead to uncontrolled overlap ratios, but will tend to drive feature overlap unrealistically high when training sets are large. This year's shared task provided an algorithm to make splits more uniformly with respect to overlap types, and it is recommended that future tasks also control for and separately analyze overlap types.
Second, both lemmas and inflectional categories are sparsely distributed in natural language use. As a result, systems in use in the real world will likely be asked to produce inflections for which lemmas or feature sets were not previously attested in their training. As focus grows on low-resource languages and language revitalization, a wide range of morphological typologies, including polysynthetic systems, will have to be reckoned with. The ability to generalize to unseen feature sets will become increasingly critical. Yet, there is a general weakness in generalization across inflectional categories in today's systems. Every system showed serious performance degradation. This was even true for agglutinative languages. Nevertheless, systems do appear to have generalized to unseen feature sets to a significant degree, and CLUZH and UBC, which showed similar overall performance, differed in their ability to handle unseen feature sets in particular. Thus, we believe there is reason for optimism and that there are real-world performance gains to be had by further developing this type of generalization.
Pushing the limits of low-resource morphological inflection.

A Lemma and Feature Overlap in 2018
Under the hypothesis that systems struggle at generalization to novel lemmas or feature sets, the proportion of test items which are novel should serve as a performance ceiling. Tables 2-3 show an apparent ceiling effect for two closely related highly agglutinative languages, Turkish and Azeri. This appendix provides performance and ceiling numbers for both lemma and feature overlap for the best performing system on each language on the low training size condition in the 2018 inflection task (Cotterell et al., 2018). This condition was chosen for illustration because it showed the most language-to-language variation in overlaps. Tables 10-11 show a ceiling effect for feature overlap in the low training condition in 2018 task. The best systems manage to surpass the hypothesized ceiling for only 17 of 104 languages, most of which are agglutinative. In contrast, lemma overlap, shown in Tables 12-13, does not seem to produce a ceiling effect. The best systems surpass it for 74 of 104 languages, which can only possible if the systems possess a significant ability to generalize to unseen lemmas.

B Full Results by Language
This section provides performance breakdowns by overlap type for each individual language for both small training (Tables 14-16) and large training (17-18) conditions. Data partition sizes can be found in Table 5.  Table 11: Difference between proportion of 2018 test set items with feature overlap and best performance in the low training condition (Macedonian-Zulu). Bolded rows indicate better percent correct than percent overlap.

C Performance by Part-of-Speech
This section provides performance breakdowns by part-of-speech for both small training (Tables 19-22) and large training (Tables 23-26) conditions. Information on the four most common parts-ofspeech in the data overall: verbs V, nouns N, adjectives ADJ, and participles V.PTCP is provided. Results for TüMorph-FST are provided separately in Table 27.