{'fraction_non_alphanumeric': 0.03218562874251497, 'fraction_numerical': 0.017623843222645617, 'mean_word_length': 5.196037099494098, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Constructing a Derivational Morphology Resource with Transformer Morpheme Segmentation

This paper describes a framework for the creation of new derivational morphology databases for a selected set of productive affixes in English. The sample resource obtained comprises almost 120k English words with morpheme segmentations generated by Transformer. The model and the database have been compared against other existing solutions. Moreover, this study offers an overview of potentially problematic cases encountered during the process of automatic word segmentation.


Introduction
Derivational morphology studies the formation of new words (lexemes) "rather than forms of a single word (cf. inflection)" (Bauer, 2004). The most common way of deriving new English words is affixation, which involves combining potential bases with affixes so that a new, morphologically complex word can be built. In the present study, two kinds of affixation are considered: suffixation (suffixes are the affixes placed after a base) and prefixation (prefixes precede a base). Affixes, as well as the bases, can be subsumed under morphemes, which are the smallest meaningful morphological units of a language (Hockett, 1958). Morphological segmentation divides words into morphemes, hence automatic morpheme segmentation employs computational methods of morpheme boundary identification. The main focus of this paper is canonical segmentation, first introduced in Cotterell et al. (2016b). It analyses a word as a sequence of canonical morphemes representing the underlying forms of morphemes, which may differ from their orthographic representations. For example, the canonical segmentation of the word funniest is fun-y-est. In principle, canonical morphological segmentation constitutes a useful, though insufficient, tool for the analysis of morphologically complex words. In this work, methods of automatic morpheme segmentation are reviewed with the aim to create new morphological resources. Initially, a machine learning model is trained to perform canonical morphological segmentation. Subsequently, English words consisting of more than one morpheme are selected for further analysis. All the model input words are potentially affixed, i.e. they contain one of the affixes (prefixes and suffixes) under review. This study also investigates how the trained segmentation model would deal with problematic morphological cases.

Related Work
Several recent studies have focused on automatic morphological segmentation. The log-linear model proposed in Cotterell et al. (2016b) is to learn to segment and restore orthographic changes jointly. In Kann et al. (2016), a character-level model consisting of five encoder-decoders is introduced and has become the new state-of-the-art. Convolutional neural networks have been applied in the process of morphological segmentation of Russian words in Sorokin and Kravtsova (2018). A discriminative joint model for canonical segmentation, with a context-free grammar backbone, has been introduced in Cotterell et al. (2016a). After applying it to a subset of the English portion of the CELEX data (Baayen et al., 1996), an annotated treebank consisting of over 7k English words was released. Importantly, Mager et al. (2020) propose two new approaches to obtaining canonical segmentations of words whilst working with limited training data: an LSTM pointer-generator and a neural transducer trained with imitation learning. The two recommended methods outperformed baselines in the low-resource setting while achieving scores close to the best models in the high-resource cases. Another attempt at generating canonical segmentations of lexical items from low-resource languages is described in Moeng et al. (2022), where Transformer obtained not only the highest performance score but also the supervised models outperformed the unsupervised ones. On the other hand, a novel, semi-automatic method of the construction of wordformation networks, focusing mainly on derivation, is proposed in Lango et al. (2021), where sequential pattern mining is used in an unsupervised manner to construct morphological features.
The application of neural networks in different computational morphology tasks, such as morphological segmentation, is delineated in Liu (2021). A model capable of building better word representations for morphologically complex words is proposed in Luong et al. (2013), where RNNs are combined with neural language models to learn morphologically-aware word representations.
Other studies, such as Jurdzinski (2017) and El-Kishky et al. (2019), show that performing morpheme segmentation may facilitate the capturing of word properties more efficiently when creating word embeddings. Song et al. (2020) demonstrate that adopting Transformer (Vaswani et al., 2017) to process morpheme information on the input layer may improve performance in the semantic textual similarity task. Hofmann et al. (2021) examine how the input segmentation of BERT (Devlin et al., 2018) affects its interpretations of derivationally complex words and suggests afterwards that the generalisation capabilities of pretrained language models could be improved if a morphologicallyinformed vocabulary of input tokens has been applied. Hofmann et al. (2020) focus on productive derivational morphology and indicate that pretrained language models, BERT specifically, could generate correct derivatives in a sentence cloze task.
Although many modifications to the standard Transformer architecture have been proposed since the original paper was published, many of them failed to do well across different applications, as demonstrated in Narang et al. (2021). Some Transformer implementations aim explicitly at improving model efficiency. For instance, Primer (So et al., 2021) achieved a smaller training cost thanks to squaring ReLU activations and adding depthwise convolution layers in self-attention. As per Wu et al. (2021), the batch size was crucial in the performance of Transformers on character-level tasks, and with a large enough batch size, recurrent networks are outperformed.
This paragraph presents several recent studies that have attempted to create morphological resources. For instance, Universal Deriva-tions constitutes a collection of harmonised (converted into a common file format and partially converted to a shared schema) word-formation resources (Kyjánek et al., 2020), while DErivBase is a rule-based framework for inducing derivational families for German (Zeller et al., 2013). That approach is further developed for Russian in De-rivBase.Ru (Vodolazsky, 2020), whereas almost 70k English words were gathered in the derivational database named MorphoLexEN and pre-

Experiments
A transformer model 1 consisting of encoding and decoding blocks was used to obtain word morpheme segmentations. The encoder block comprised positional embedding, multi-head attention, feed-forward and dropout, while the decoder blocks were constructed with the same layers, but the positional embedding layer was masked. The Transformer implementation used for experiments differed slightly from the one proposed in Vaswani et al. (2017). Learned positional encoding was applied instead of a static one, the optimiser's learning rate was static instead of one with warm-up and cool-down, and no label smoothing was utilised. The implementation of the model was inspired by that explored in Moeng et al. (2022). The hidden dimension was set to 256, and the learning rate worked best at 0.0005. A relatively small dropout of 0.1 was applied. Various optimizers available in PyTorch were tested, e.g., Adam (Kingma and Ba, 2014), RAdam (Liu et al., 2019), NAdam (Dozat, 2016), AdamW (Loshchilov and Hutter, 2017), Adadelta (Zeiler, 2012) and Adagrad (Duchi et al., 2011). Adam was chosen in Vaswani et al. (2017) and Moeng et al. (2022), but AdamW led to slightly better results in BERT. NAdam performed best in this research. Different activation functions were tested to replace ReLU (which was used in Moeng et al., 2022), and even though the differences in the model scores obtained were not significant, consistently, the best results were obtained with GeLU (Hendrycks and Gimpel, 2016). Squaring ReLU activations, as proposed in Primer slightly decreased performance which decreased even more after trying out Swish units (Ramachandran et al., 2017).
The new derivational morphology resource was built with Transformer word morpheme segmentation. The model was trained on the data from MorphoLexEN. The words used to develop this resource were obtained from the English Lexicon Project (Balota et al., 2007) and were already segmented into morphemes. Inflectional suffixes such as -s, -ing or -ed and contractions such as 'll or 's were removed manually. Out of 68,624 words in the database, 80% formed the training set, and 10% were assigned to validation and test sets.
A relatively extensive list of English words was compiled out of lexical items from various sources: NLTK corpus (Bird and Loper, 2004), Brown corpus (Francis and Kucera, 1979) and built-in English word lists of macOS and Ubuntu. Each word was case-insensitive. Many words overlapped, so all the duplicates had to be removed. Then, all the individual lists were merged into one list containing 315,404 words. Finally, each word from the list was automatically segmented and entered in the morphological resource, provided that the relevant number of automatically segmented morphemes was greater than one and the lexical item under study started or ended with one of the selected affixes. A set of recognisable productive affixes considered in this study is presented in Table 1.

Model
Accuracy F1

Results
In this section, model performance is compared to other solutions, the new derivational morphological resource is evaluated, and puzzling morphological cases are analysed.

Model performance
The model used to create the morphological resource was trained on the subset (Cotterell, 2016) of the English portion of the CELEX lexical database with the view to compare model performance with other modern solutions. The reported results were obtained with 10-fold crossvalidation. The training, validation and test sets consisted of 8k, 1k and 1k samples, respectively. Encoder and decoder dropouts were increased to 0.3 to account for limited data issue. Adam optimization and ReLU activations seemed to work best in this low-resource setting. Two metrics were used for comparison: accuracy and morpheme F1 (Van den Bosch and Daelemans, 1999). Segmentation accuracy measured whether every canonical morpheme was identified correctly. This implies that this metric is very harsh, and very close answers are penalized equally as the wrong ones. Morpheme F1 would give credit only if some canonical morphemes were identified correctly. Results are exhibited in Table 2, where the developed model was compared with Semi-CRF (Sarawagi and Cohen, 2004), Joint (Cotterell et al., 2016b) and Joint+Vec (Cotterell and Schütze, 2018).
The Transformer accuracy and F1 measure are close to the scores of other models. More data would probably significantly increase the performance of the tested model. The model used to create the new resource was trained on a several times larger dataset (a subset of MorphoLexEN) and achieved over 94% morpheme F1 and almost 93% segmentation accuracy on the test dataset.

The new resource
The obtained morphological resource, named Morfem, consists of 118,900 words supplied with their segmentations 2 . In what follows, the evaluation of the database is discussed. One thousand random words from the database were manually checked to determine whether their morphological status was correctly recognised. It turned out that over 90% of the randomly selected words constituted complex words derived with one of the selected affixes. The words which were manually marked as simplex yet segmented by the model could be subsumed under different categories. The general list included some proper names, e.g., Demontez was segmented as De-montez, along with lexical items that were not listed in English dictionaries, e.g., unie, or, misspelled words, e.g., tecnology. Some morphological cases appeared to be problematic. Certain primarily lexicalized words with potentially divisible internal structures may pose some obstacles, e.g., the words delay and discard may be treated either as delay and discard or de-lay and dis-card. In MorphoLexEN, delay was treated as a single morpheme, while discard was divided. The model managed to learn that, and thus only discard was included in the resulting database (was divided into dis and card).
To automatically validate the resource, 901 derivatives containing one of the affixes under study were retrieved manually from Joseph Conrad's Heart of Darkness (Conrad, 1899(Conrad, /2006 set of derivatives from the book, were marked as true positives. Words that were present in the book and a database, but not in the manually selected set were counted as false positives. Finally, the words that were manually selected, but not found in a resource were designated as false negatives. The test results are presented in Table 3. Two versions were compared with the other databases. One with strict matching, where a word was noted in the resource only if one of the identified morphemes overlapped with an affix from the list. The other, without strict-matching, included all the words which contained more than one morpheme, and started or ended with at least one of the selected affixes. Morfem with strict matching achieved the highest precision while lacking in recall. Morfem with nonstrict matching achieved the highest coverage of the derivatives, which is indicated by the highest recall score among the compared databases. Combining the non-strict Morfem with other resources (excluding the strict-matching Morfem) to form a unified vocabulary resulted in even higher recall alongside a significant precision decrease. Deciding which metric is the most relevant depends on the specific application.

Conclusion
The proposed framework allows for creating morphological resources larger than those currently available. The automatic morpheme segmentation task results are promising, but there is still some room for improvement. Therefore, a more reliable linguistic resource could be compiled when built upon a more reliable segmentation algorithm. Current state-of-the-art methods of canonical morphological segmentation do not consider the word's context. Knowing that words can be divided differently depending on their context (e.g., recover or re-cover), methods consulting the context should be developed.