{'fraction_non_alphanumeric': 0.02671624320044068, 'fraction_numerical': 0.013289265303311988, 'mean_word_length': 4.638198757763975, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Assessing Digital Language Support on a Global Scale

The users of endangered languages struggle to thrive in a digitally-mediated world. We have developed an automated method for assessing how well every language recognized by ISO 639 is faring in terms of digital language support. The assessment is based on scraping the names of supported languages from the websites of 143 digital tools selected to represent a full range of ways that digital technology can support languages. The method uses Mokken scale analysis to produce an explainable model for quantifying digital language support and monitoring it on a global scale.


Introduction
The users of endangered languages struggle to thrive in a digitally-mediated world. The opportunities afforded by digital technology differ drastically depending on the language being used. This has been dubbed the "digital language divide" (Mikami, 2008;Young, 2015;Soria, 2016;Matsakis, 2019). As digital modes of communicating and accessing information become increasingly necessary in daily life, lack of digital language support (DLS) for a language means that its speakers must use other languages to participate in the global information society or be left out.
However, in a world where most people are multilingual and each language fits into its functional niche within an ecology of languages (Lewis and Simons, 2016), full digital support for every language is not a realistic goal nor what those multilingual individuals are necessarily looking for (Bird, 2022). The goal of our research is to develop a method for measuring DLS in every language, so that it will be possible to provide an empirical view of the digital state of the world's languages and to observe the progress as so-called low-resource languages move toward crossing the digital language divide.

Related Work
Our primary inspiration has been the seminal work by Kornai (2013) on developing a method for assessing the digital vitality of any language. He proposes a four-way classification of languages as digitally Thriving, Vital, Heritage, or Still, "roughly corresponding to the amount of digital communication that takes place in the language." His method harvests data from the Web, then uses supervised classification to automatically label all known languages. In practice, he adds a fifth level, Borderline, to represent languages that show signs of crossing the gap from Still to Vital. He and his colleagues have applied this method to the languages of India (Kornai and Bhattacharyya, 2014), the former Soviet Union (Kornai, 2015), and the Uralic family (Acs et al., 2017).
In reviewing Kornai's method, Gibson (2015Gibson ( , 2016 focused on the huge gap between Still and Vital. He argues that two additional levels are needed to fill this gap: one for when the needed elements (like a keyboarding solution) are in place for potential digital language use, and another for when digital language use is indeed taking off. We follow Gibson's lead in adding two levels, but use names that achieve better congruence with the geometry of the S-curve model that emerges from our method (see Figure 3).

Requirements
Following Kornai's (2013) lead, we seek to develop an automated method for assessing digital language vitality that is based on feature data harvested from the Web. In this way, it can be run periodically to monitor changes in digital vitality for every language. We were motivated to develop an alternative to Kornai's method of analysis in order to meet three requirements: Digital vitality should be orthogonal to nondigital vitality. We exclude features like population and language vitality from the feature data. Kornai notes that the EGIDS level as reported in Ethnologue (Lewis and Simons, 2010;Eberhard et al., 2022) is "the best predictor of digital status." But digital vitality is distinct from non-digital vitality. For instance, our method reports the "dead" language Latin to be the 80 th most digitally vital language in the world. By contrast, Aimaq with nearly two million speakers is found to be digitally Still.
The assessments should be explainable. A standard critique of machine learning models based on black-box methods is that the models cannot explain why they produce the answers they do (Arrieta et al., 2020;Miller, 2019). Kornai (2013) bases his results on the majority outcome from 100 runs of a black-box model that yields a slightly different result each time. Users will be more likely to trust results if they are deterministic and explainable.
The assessment scale should measure a single underlying trait. The data features used by Kornai (2013) covered a variety of digital uses. Some had to do with quantifying the extent to which the language has been documented in digital archives by researchers. Others, like the sizes of Wikipedias, had to do with quantifying the extent of digital language use by the language community itself. Still others looked at specific software products and recorded which languages they support. These strike us as three distinct traits, each of which should be assessed in its own right: digital language preservation, digital language use (DLU), and digital language support (DLS). Of these, the latter two are what speak to monitoring the digital vitality of a language as it moves toward crossing the digital language divide. DLU and DLS are distinct traits that should be assessed separately-speakers of unsupported languages may nevertheless use it digitally (for instance, making do in texting; see Eberhard and Mangulamas (2022)), while speakers of supported languages may choose to use digital resources in another language they know.
We have chosen to focus on DLS since the data for monitoring that phenomenon are openly accessible-the developers of digital tools are usually keen to advertise all of the languages they support. By contrast, data on actual digital use is typically not shared on a language-by-language basis by the vendors concerned. A comparable effort to assess DLU on a global scale is much needed, though we anticipate that it will be significantly harder to acquire the needed data.

Methodology
The method we have adopted for building an explainable model of DLS is Mokken scale analysis (Mokken, 1971;Schuur, 2003). Mokken's method is a generalization of the more widely known Guttman scaling (Guttman, 1950). In the latter, the items in a scale form a strict hierarchy. If a subject has an item on the scale, then all lower items also apply. A subject's score on the scale is thus the highest item that is true for the subject.
Intuitively, DLS has these properties. If a language has a good virtual assistant (like Siri), then we can infer that it also has good machine translation-but having good machine translation does not imply having a good virtual assistant. Similarly, if a language has good machine translation, we can guess that it must also have good spell checking, though we cannot assume that the reverse would hold. In a Guttman scale, an exception to the hierarchical ordering is considered an error, but in an arena like DLS we can expect there to be exceptions. Mokken scaling is a method for placing the items of a supposed hierarchical scale into their optimal order, while providing metrics that allow one to evaluate how well the hierarchical model fits.

Categories of Digital Language Support
The method uses the following seven categories of DLS. They are listed below from easiest (most commonly supported) to hardest (least commonly supported) as determined by the results of our analysis: 1 • Content -A service offering content in many languages (like Wikipedia, news sites, or Bible sites) 2 • Encoding -A system component for representing languages (like keyboards and fonts) • Surface -A tool with surface-level processing (like spell checking or stemming) • Localized -A tool with a localized user interface (like operating system, browser, or messaging) • Meaning -A tool with meaning-level processing (like machine translation) • Speech -A tool for speech processing (like speech-to-text or text-to-speech) • Assistant -An intelligent virtual assistant (like Siri or Alexa) For each category, we sought to identify the top ten tools of its kind globally. In order to ensure that we included the major tools in use outside the English-speaking world, we also included the top five tools in each of the ten most populous countries of the world. 3 The reference authority for these rankings was the similarweb service. 4 Then we added any tools found from other sources that supported more than 10% of the median number of languages supported by the top tools in the category. In order for a tool to be used in our analysis, we required there to be a URL from which the names or ISO 639 codes of supported languages could be scraped.
The full sample consists of URLs for 143 digital tools across the seven categories of DLS. 5 The number of tools in each category is shown in Table 1 as the maximum number in the range for level 4.

Harvesting the feature data
The method works by scraping each URL in the sample to discover what languages each tool supports. The harvested language names are mapped to their corresponding ISO 639-3 code 6 by means of a manually maintained table of name-to-code mappings. After the mapping of the harvested language names, the resulting feature data is a logical matrix with rows for 7,829 ISO 639-3 codes, columns for the 143 digital tools, and a Boolean value at the support for the language. To measure digital language use, we would quantify the amount of digital content in each language.
3 This sampling method allows us to discover widely-used tools that support just one large language, but it admittedly misses tools that have been custom-built for a single smaller language. 4 https://similarweb.com 5 A complete list of the 143 digital tools is provided at https://github.com/sil-ai/dls-results. 6 https://iso639-3.sil.org/code_tables intersection indicating whether the given language is supported by the given tool.

Scoring the DLS categories as subscales
When a language is not supported by any tools in a given DLS category it is scored as 0; otherwise, the number of tools supporting that language is converted to a level score on a four-level subscale. The correspondence between the number of tools supporting the language and the level on the subscale is shown in

From category levels to scale items
In constructing the Mokken scale, the levels of the categories become items in the scale. These items are named Content1, Content2, and so on. Within each subscale, the items form a strict hierarchy, in which being scored at a higher level on the subscale implies also having at least as much support as the lower levels of the same subscale. Thus the count of languages for item Content3 also includes the languages for Content4, and so on going down. The bar graph in Figure 1 shows the items listed from top to bottom in ascending order of the number of languages with at least that level of support in the named category.

Evaluating fit of the model
Mokken scale analysis allows us to evaluate the degree to which the scale depicted in Figure 1 forms   The results in Table 2 show that the proposed DLS scale is a very strong scale, especially among the categories of support that are hardest to achieve. Thus the total score on all 7 categories (i.e., 0 to 28) serves to quantify the DLS for a given language.

Relative difficulty of DLS items
Mokken analysis is based on Item Response Theory (IRT)-a methodology developed for educational and psychological testing (Lord, 1980). In IRT, logistic regression is used to derive an Item Response Function (IRF) for each test item; it returns the probability that a subject would produce a positive (or correct) response on that item, given their total score on the rest of the test items. The difficulty of an item is defined as the score (on the rest of the test) at which the subject has a 50% chance of giving a positive response for the item. Figure 2 plots the difficulty for each of the scale items listed in Figure 1. For instance, a language has a 50% chance of getting its first spell-checker (Surface1) if it has 3.6 other DLS items, but the first virtual assistant (Assistant1) cannot be expected until it has 23.4 other DLS items.  Figure 3 plots the DLS score for 7,829 ISO 639 languages. The vertical axis is the measure of DLS as a proportion: the DLS score achieved divided by the maximum possible score. 9 The horizontal axis is the rank of the language by DLS score, but converted to a log scale and flipped so that lowest DLS is on the left and highest is on the right.

DLS as a growth curve
The pattern that emerges is an S-curve as is typical in studies of growth in innovation. We follow the geometry of the fitted curve to assign each language to one of the five summary levels: • Still -a score of 0 • Emerging -at the bottom where the slope is more horizontal than vertical • Ascending -below the midpoint where the slope is more vertical than horizontal • Vital -above the midpoint where the slope is more vertical than horizontal 9 The DLS scores are also adjusted by scoring each item as the probability returned by its IRF. In educational testing, scoring each positive response as a probability is a way of controlling for random guessing on questions that are too hard for the subject. In the application to DLS it can control for "random" developments that do not have the underpinnings of the expected lower categories of support, such as when there is a one-time philanthropic gesture by a large company or the potentially unsustainable efforts of a solitary developer. • Thriving -at the top where the slope is more horizontal than vertical By comparing Figures 2 and 3 one sees what components of DLS correspond to the summary levels. Table 3 reports the number of languages at each summary level along with the names of example languages, the first being from the upper end of the range and the second from the lower. 10

Level
Languages

Conclusion
We have presented a method that produces an explainable model for quantifying DLS. We are currently working with Ethnologue to add reporting on DLS in its description of languages, beginning with the next edition. Regularly updating the assessments should serve to document the digital trajectory of every known language.
10 A sampling of the detailed results produced by the system is provided at https://github.com/sil-ai/ dls-results.