{'fraction_non_alphanumeric': 0.022551899238313338, 'fraction_numerical': 0.011699108876387714, 'mean_word_length': 5.152526799387442, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Processing M.A. Castr\'en's Materials: Multilingual Typed and Handwritten Manuscripts

The study forms a technical report of various tasks that have been performed on the materials collected and published by Finnish ethnographer and linguist, Matthias Alexander Castr\'en (1813-1852). The Finno-Ugrian Society is publishing Castr\'en's manuscripts as new critical and digital editions, and at the same time different research groups have also paid attention to these materials. We discuss the workflows and technical infrastructure used, and consider how datasets that benefit different computational tasks could be created to further improve the usability of these materials, and also to aid the further processing of similar archived collections. We specifically focus on the parts of the collections that are processed in a way that improves their usability in more technical applications, complementing the earlier work on the cultural and linguistic aspects of these materials. Most of these datasets are openly available in Zenodo. The study points to specific areas where further research is needed, and provides benchmarks for text recognition tasks.


Introduction
As a research domain, the Natural Language Processing has regularly focused on the formal written varieties of the most widely used languages of the world. At the same time there has been a growing interest in both non-standard and informal language Partanen et al., 2019), and their historical varieties (Säily et al., 2021;. The research potential of historical language varieties is clearly on the upbound, and one can argue that the need is already quite evident, as digitization processes in libraries and archives around the world have reached relatively mature stages and already have large digital collections available. Finnish ethnographer and linguist Matthias Alexander Castrén (1813Castrén ( -1852 produced a large collection of field notes, and also published widely on languages of Northern Eurasia. Recently, two hundred years had passed since his birth, and in this connection the Finno-Ugrian Society launched a project where several of his field notes and grammars are published as commented editions, available both digitally and in print. Numerous monographs have already been published in the series (Salminen et al., 2020;Lehtinen, 2017;Salminen, 2017;Forsberg, 2018;Häkkinen, 2019;Salminen, 2019;Salminen and Janhunen, 2021). The complete series will contain more than twenty volumes. This article discusses the processing of original raw materials up to this point, with a goal of setting a vision of how this process can be refined later on.
Within the research tradition of the Uralic languages, Matthias Castrén is often renowned as the most significant Finnish linguist of the 19th century. Castrén collected vast materials from almost thirty languages on his expeditions to Lapland and Northern Russia between 1838 and 1849 (Janhunen, 2017(Janhunen, updated 2021. The materials are stored in the National Library of Finland. The number of handwritten manuscript pages is approximately ten thousand. Castrén's work carries a unique historical dimension for the languages he studied, and his manuscripts and extensive correspondence with other researchers of the time are also valuable for the history of scientific research.
Our study presents individual datasets built from Castrén's materials and reports benchmarks on various text recognition experiments. The main repository for related data is Manuscripta Castreniana collection in Zenodo 1 , and other locations are specified when datasets are discussed. We also discuss individual experiments with the text recognition of Castrén's unpublished and published materials, and contextualize the results more widely within early linguistic descriptions. We analyse some of the challenges met in further processing of this content, and delineate possible ways forward. The most important step we can identify is making these materials better available, so that further work can build upon the contributions of more researchers. This is also the step we are trying to help make. We would hope, for example, that eventually Castren's materials would be included in different shared tasks. In the same spirit we also share all our processing code in GitHub 2 , which we hope makes these materials easier to access for different researchers in the digital humanities and related fields.

Related work
Historical dataset creation is one topic that connects closely to ours. Especially within the Universal Dependencies project (Zeman et al., 2021) there are numerous instances of historical language treebanks. There are five Latin treebanks, Old Church Slavonic treebank (Haug and Jøhndal, 2008), Old Turkish (Derin, 2020) and Old French treebank (Stein and Prévost, 2013), just to mention some of them. Such resources are in a central role, as they allow training NLP models to address different downstream tasks for these language varieties. Naturally, any openly available resource, in plaintext or with annotations, can be used for these purposes. At the same time, the CoNLL-U file format offers a good and well understood structure that can easily be compared.
There are examples of such datasets being used in downstream tasks, such as lemmatizers and POS taggers created for Latin (Clérice, 2021) and Old French (Camps et al., 2021). Work has been done also on Old Swedish (for example, (Borin and Forsberg, 2008;Adesam and Bouma, 2016), but an actual diachronic corpus seems to be still under construction (Pettersson and Borin, 2019). If such resources existed, the analysis of Castrén's 19th century Swedish would be in a different state. There is one unannotated diachronic corpus of Old Literary Finnish (Institute for the Languages of Finland, 2013) and one morpho-syntactically annotated corpus of Mikael Agricola's works (Institute for the Languages of Finland and University of Turku, 2020). The latter has already been used to develop a lemmatizer as well . Named entity recognition (NER) for historical publications in Finnish has also received attention lately . A recent survey by Humbel et al. (2021) reviewed different named entity recognition systems for early modern textual documents. Their conclusion was that benchmarking different NER systems in this domain is not currently possible, and suggest wider use of shared forums such as computational linguistics conferences as one way to coordinate further discussion and practices. Study by Idziak et al. (2021) where Polish lexicographic cards were recognized and organized is in some aspects also close to what we would hope to achieve with materials discussed here. To our knowledge, there are no datasets, NLP tools or resources of historical varieties of the endangered languages included in these collections, especially in Castrén's writing system that is essentially an inauguration of a Latin based transcription (Latin transcription with some Cyrillic characters).

Materials
We discuss four sections of Castrén's materials. The first consists of ethnographic field notes in 19th century Swedish under the title Ethnographiska, historiska och statistiska anmärkningar. Castrén wrote these texts in a extensive area that belongs to the northern regions of the contemporary Russian Federation. This text is also multilingual, with numerous expressions in Cyrillic, but we can approach it largely as a Swedish text. This subset contains 188 pages of handwritten texts. We use this dataset in text recognition experiments reported below, but these materials will be added to the Zenodo collections at a later stage.
The second dataset comes from Tundra Nenets epic poems that have a Russian translation with Swedish commentary. The Figure 1 displays the typical structure in this manuscript. The page is split into two loosely distinguished columns, with Tundra Nenets transcription on the left and the Rus-sian translation on the right. In the upper right region we see a comment in Swedish in parentheses, but there are also parenthetical clarifications in Russian, as seen in the bottom right corner. All in all, the material comprises 192 pages. This example also provides a good illustration of how the layout detection of these manuscripts is an additional challenge. This dataset is published as is in Zenodo (Castrén, 2021b). The texts have been aligned line by line into the microfilm scans of the original manuscripts in collaboration between the University of Innsbruck and the Finno-Ugrian Society, and this material is an excellent test set for various tasks including text to image alignation, line segmentation and handwritten text recognition.
The third dataset contains published Komi-Zyrian grammar of Castrén (Castrén, 1844) that is written in Latin. The grammar is 174 pages of printed text, all together. In the Manuscripta Castreniana project an English translation with commentary will be published, which adds a new dimension to what kind of computational tasks could be studied with this collection. Additionally, this partially proofread dataset is located in Zenodo (Partanen and Rueter, 2021), with 26 proofread pages of which 3 contain manually constructed tables. Thereby, this dataset is an example of 19th century printed Latin linguistic description, but also serves as the ground truth data for table layout detection as several tables are included with defined table cell structure. This grammar is also available as two different scans, both archived in Zenodo.
The fourth dataset contains Castrén's Komi-Zyrian wedding laments and their transliteration in the modern Komi orthography . These materials were published with Finnish and German translations by Aminoff (1880), and our dataset contains aligned versions of the translations and different transcriptions. Similar dataset could also be created from Castrén's translation of the Gospel of St. Matthew. Crucially, Castrén's transcription system cannot be automatically converted into current orthography as it does not contain all phonemic information that the orthography does. However, the dataset, in itself, is very illustrative of a wider problem in applying NLP to these kinds of materials: the textual representation used has such a different level that, if we cannot transform the transcription into a more modern writing system, we cannot access the text with any current tools.

Text recognition 4.1 Background
Text recognition of historical handwritten documents has advanced rapidly in the past few years. The Transkribus platform (Kahle et al., 2017) is leading the field in usability and adoption, and there are reports of consistent results. These include materials by authors such as Foucalt (Massot et al., 2019), Eugène Wilhelm (Schlagdenhauffen, 2020), Jeremy Bentham (Muehlberger et al., 2019, 959) and Konstantin Rychkov . As mentioned, Castrén's texts include dozens of languages, and Russian, Swedish and Latin are all used as meta languages in different contexts. Similarly presence of different writing systems is also a feature, and challenge, of datasets mentioned above, both with mixed Latin and Greek characters (Schlagdenhauffen, 2020, 4) and Evenki-Russian mixed content . However, the wide array of endangered languages is still a very specific feature of Castrén's materials.
Currently all text recognition experiments with Castrén's data have been done using the Transkribus platform. The reason for this has been that it allows collaborative editing, and has, at least for handwritten materials, been the currently leading platform. In our further processing the data from Transkribus is exported in Page XML format, which in our experience has been very satisfactory. It appears that Castrén's materials are still particularly challenging to process, and we aim to delineate some of the more technical reasons next.
The first part of the materials was aligned with the microfilm images from XML files where one unaligned transcription version already existed. As these transcriptions were done outside Transkribus, with no visual connection to the actual documents, there may be features in the transcription that should be revised. At the same time the transcriptions were done before the text recognition task was even possible, so the character choices were primarily based on what was convenient for the individual researchers. When tens of thousands of pages are analysed together, it would be important to give careful consideration to which characters should be used to represent which of Castrén's special characters. This work is partially technical and a matter of deciding the correct Unicode characters, but also relates to linguistic analysis. The analysis of the latter type was also conducted for Evenki by Arkhipov and Däbritz (2021). As these early versions have been aligned with microfilm scans, and only later have the better quality versions been scanned from the original documents, it may become necessary to realign the transcriptions with these more accurate versions. The materials have been arranged so that such a task is in principle feasible. The second dataset discussed in this study contains exactly these aligned microfilm scans, which, we believe could be used to measure both the impact of chosen character conventions and the quality of scans to the recognition result. The higher quality images are also stored in Zenodo with extensive metadata about the page content (Castrén, 2021a). Generally it is very typical for Castrén's materials that the same text exists in multiple versions. It is unclear to the current authors how to best connect these versions, but we see potentially high value in such an undertaking.

Experiments
In the current workflow, all texts are manually verified. The ground truth material increases continuously, and has now reached 358 pages. This includes 19,490 lines and approximately 57,000 words. The text recognition accuracy has not significantly improved when the last hundred pages have been added, and the accuracy has been hard to improve further. We first discuss the results with Castrén's printed materials shown in Table 1 and then discuss the handwritten text recognition results shown in Table 2.
Castrén's Komi-Zyrian grammar is written in Latin and it contains individual Komi words and experssions plus some comments in Russian. As Transkribus already contains numerous text recognition models for printed texts, the ideal scenario would be to use some of these directly. We compared some of the Transkribus models for printed texts against the proofread materials, the result being presented in Table 1 Although Transkribus print 0.3 model 3 does not even include Latin, it still performs extremely well in our test scenario. In the model's documentation CER of 1.6% is reported, and in our experiment the result was even better than that. This has wide significance for work on printed Latin texts, as the out-of-the-box tool truly gives functional result. This should be taken into account when planning further work on printed materials. As expected, the Russian words did not get recognized, and the printed model could benefit from wider inclusion of scripts.
With the handwritten materials the situation is different. We can see in the Table 2 that none of the available HTR models for the Swedish language work very well, even though the result on the Count Records model from the National Archives of Finland is relatively good. As this model is contemporary with Castrén, and also contains handwritten Swedish, the accuracy is not necessarily surprising. Yet, it tells that even with a handwritten text recognition model we do not need to start entirely from scratch.
Even if we were to try to use other models as base models in training, the gains would be relatively minor.  We are not seeing entirely transformative differences in the results, but still there is a significant improvement that we get essentially for free.

Processing tools
We have archived our processing scripts on GitHub and Zenodo so that they would be maximally useful for a wider community of researchers. Text recognized materials from Transkribus can be exported in Page XML format. The structure is highly standardized, but also relatively complicated. We provide methods to read the lines and their bounding boxes from the XML files into a Python dictionary. After this different operations can be applied, but at a different level: there are already many packages often provide deeper language specific functionality that should be leveraged. Example include UralicNLP (Hämäläinen, 2019) for basic NLP analysis of Uralic languages, and murre for specific dialectal and historical text normalization or lemmatization scenarios (Partanen et al., 2019;Hämäläinen et al., 2020). The NLP for Latin also seems fairly developed, and available models could be applied (Clérice, 2021). We see as specific challenges in this the multilinguality and the continuous presence of words and expressions in different languages.

Further usage
The materials we have discussed have been created for two purposes: 1) openly licensed ground truth material for text recognition models, and 2) recognized, manually corrected, text for ethnographic and linguistic research. Text recognition models are at the moment line-based and the latter mainly relies on the subject knowledge of the researcher. Neither of these tasks necessarily demands further automatic processing of the materials, at least as long as the research is based on visual use of original versions and the recognized text is used as an aid and search tool to navigate in the document.
However, we consider it still extremely important to be able to extract the text correctly from the files. In the current dataset both the line and layout element structure is indicated by order numbers, and simple concatenation of the lines thereby, in principle, yields the wanted order. However, there are cases where the situation is more complicated. The running order of the elements and lines may not be manually corrected, and it relies on individual conventions whether there is some way to mark whether the order has been verified manually.
In our dataset we find that the running order of the text is generally correct at the page level, and especially so in document pages where layout is simple. This includes the printed Komi-Zyrian grammar and ethnographic notes. In the former table layout detection would need attention, and in the latter problems arise primarily from margin notes and comments between lines. Currently those are not easily placed to the correct locations in the text. In complexly layouted documents with several columns we also find a question of how to indicate the relationship across the columns, as one line is often translation or comment of the other. This issue is seen on almost all pages of the Tundra Nenets epic narrative dataset.

Conclusion
Our experiments show that the currently available tools to process 19th century Latin grammar materials in an endangered language can be almost flawlessly recognized with out-of-the-box text recognition models. With handwritten materials the publicly available models need to be customized, but the current accuracy may give at least some starting point if the language and time period match. The divergence of transcription systems and their complex relation to the contemporary orthographies is one challenge that needs to be separately addressed.
To advance actual NLP applications, we also suggest that a sample from Castrén's materials would be published as a treebank or other annotated structure. Such multilingual collection may not fit larger projects such as Universal Dependencies, but similar conventions and file structures could easily be used. How this can be connected to proofread Ground Truth resources, commentaries and digital editions is another question, but there are few materials better for testing this than Castrén's data that is openly available and still acutely relevant for contemporary research.