{'fraction_non_alphanumeric': 0.03161573902451941, 'fraction_numerical': 0.011279987290155166, 'mean_word_length': 5.307281229124916, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 4, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Context-sensitive evaluation of automatic speech recognition: considering user experience & language variation

Commercial Automatic Speech Recognition (ASR) systems tend to show systemic predictive bias for marginalised speaker/user groups. We highlight the need for an interdisciplinary and context-sensitive approach to documenting this bias incorporating perspectives and methods from sociolinguistics, speech & language technology and human-computer interaction in the context of a case study. We argue evaluation of ASR systems should be disaggregated by speaker group, include qualitative error analysis, and consider user experience in a broader sociolinguistic and social context.


Introduction
Automatic Speech Recognition (ASR) has become a common tool in human-computer interaction, enabling, for example, voice user interfaces and (imperfect) automatic captioning of multimedia content. As with other language technologies (e.g. Sap et al., 2019;Blodgett and O'Connor, 2017), rapid improvements in performance have not been equal for different user groups. As Blodgett et al. (2020) show, discussions of this "bias" are often poorly defined, not grounded in explicit normative judgments and divorced from socio-historical contexts, origins and harms of the system behaviours. In this paper, we argue that researchers at the intersection of speech and language technologies (SLT), humancomputer interaction (HCI), and sociolinguistics are well-placed to consider the experiences and social context of different speaker/user groups in critical quantitative and qualitative evaluations of ASR systems. Knowledge about language variation and its relation to society coupled with expertise from HCI allows us to understand how predictive biases reflect larger social structures and ideologies about language, and how they affect users.
After presenting prior work on language variation and ASR, a case study of self-recorded au-dio diaries collected for the Lothian Diary Project 1 highlights the need for a context-sensitive approach to ASR evaluation which we outline.
2 Language variation, bias and ASR Blodgett et al. (2020)'s critique notwithstanding, predictive bias, defined here as error and outcome disparities for different user groups (Shah et al., 2020), has become a research focus in SLT and other machine learning fields as applications are extended to high-stakes contexts such as hiring, policing and banking where they have been shown to (re)produce structural inequalities (see e.g. Benjamin, 2019). Predictive bias also appears to be prevalent in commercial ASR systems for English 2 .
Recent work describes stark racial bias in commercial American English ASR systems (including Google's Cloud Speech) (Koenecke et al., 2020), with much higher word error rates (WER) 3 for speakers of African American English (AAE) than white speakers of (Californian) American English. Notably, these types of error disparities appear to be driven by under-representation of AAE training data both for the acoustic modelling (Koenecke et al., 2020) and the language model used to decode sequences of phones into utterances (Martin and Tang, 2020). "Regional" variation has also been reported as a source of unequal performance, with particularly high error rates reported on YouTube's captions for speakers from Scotland and (the US state) Georgia (Tatman, 2017). Similar to more recent work, YouTube captions have been found to perform worse for African American speakers (Tatman and Kasten, 2017). These problems are not limited to proprietary systems, as Mozilla's open source system DeepSpeech performs significantly worse for speakers of Indian English than "American English" 4 (Meyer et al., 2020), and also fails to transcribe AAE morpho-syntactic variation correctly (Martin and Tang, 2020). While some early research has suggested ASR performance differences based on (binary) speaker gender (Adda-Decker and Lamel, 2005;Benzeghiba et al., 2007;Tatman, 2017), it is unclear that gender by itself is a significant factor in recent systems (Tatman and Kasten, 2017;Meyer et al., 2020). Koenecke et al. (2020) suggest that the interaction of gender and race is significant, with differences between Black men and Black women being more significant than between white men and white women or men and women across race 5 . These results appear to be linked to speaker's speech styles (e.g. in Adda-Decker and Lamel, 2005) and use of dialect features (Koenecke et al., 2020), both of which have long been documented to pattern with gender (see Labov, 1990, for a classic paper) and could be correlated with gender in training and test sets. Other work in this space has focused on the potential of ASR to improve accessibility of audio media and digital technologies, looking at experiences of Deaf and hard of hearing users (Glasser, 2019) and dysarthric speakers (De Russis and Corno, 2019; Young and Mihailidis, 2010). For both groups commercial ASR systems perform quite poorly, though the severity and amount of errors varies by speaker. Research on predictive bias in commercial ASR for regional varieties of English beyond the United States and in the context of systems not exclusively trained on American English, as well as experiences of second language learners of English, and other groups who are potentially particularly reliant on ASR to access computing technologies such as elderly people, is sparse.
From a linguistic perspective, no language variety or speech style is inherently more difficult, incorrect, or inappropriate than any other. There are, however, powerful ideologies regarding the relative status of different varieties and styles which are rooted in broader socio-historical contexts and reflect the social status of the groups who speak them (Woolard and Schieffelin, 1994). In addition to being stigmatised in "traditional" contexts of power in society, varieties spoken by marginalised communities appear to be (not coincidentally) underrepresented in the data we use to build and evaluate speech technologies, leading to substantial predictive biases making speech technologies less accessible to already marginalised groups.

Lothian Diaries: A case study
The Lothian Diary project is an ongoing interdisciplinary research project inviting residents of the Lothians region of Scotland to contribute selfrecorded audio and video diaries about their experiences of the COVID-19 pandemic. The more than 120 diaries collected so far are highly variable in recording quality, number of speakers and topics discussed, and participants are diverse 6 in terms of age, gender, linguistic background, ethnicity, socio-economic class and level of education. Edinburgh and the surrounding Lothians region are of particular interest for sociolinguistic research because of the capital region's status as a centre for higher education, finance and tourism. In addition to the variation within Scottish English 7 between different areas and different socio-economic groups within the city, there is also a wide range of other first and second language varieties of English, as well as other languages. The Lothian Diary project also includes many of these other varieties of English, rather than focusing on speakers with long residential histories in a particular area (as is often the case in sociolinguistic work) or first language speakers (as is usually the case in SLT evaluation). The recordings form a highly naturalistic and exceptionally varied data set. ASR is used here to facilitate social science research which requires accurate and complete transcriptions (achieved through manual correction).
So far, 13 diaries submitted by participants who agreed to have them made public, have been processed with the Google Cloud Speech-to-Text API 8 (GC STT). Diaries (16 kHz FLAC files) were processed in their entirety using the model used for long audio files which uses asynchronous speech recognition. WER was computed separately for each speaker using sclite 9 . In the following section, we present a brief qualitative error analysis. WER for individual speakers varies dramatically (see Table 1). Some of these errors appear to be related to accent differences. For example, Scottish speakers' pronunciations of I or I've are frequently mistranscribed as ah or of and other accent-based errors include: cat [kaP] > car, living > leaving, hating our > heating are. However, there is also significant variation within each accent group. GC STT fails to transcribe filled pauses (uh, um) and word fragments and occasionally deletes false starts and repetitions. Furthermore, errors appear to be more prevalent in the vicinity of hesitations and repetitions. As a result speakers who produce more hesitations and repetitions tend to have higher error rates, while people who appear to read from prepared notes tend to be more fluent and have lower error rates. The highest WER in this sample derives from a recording by a Scottish English speaker who produces many false starts, word fragments and a number of Scots words (which the system likely would not recognise under any circumstances). Words are also often substituted by a wrong (but often grammatically appropriate) inflectional form (e.g. past tense > present tense).
All of these errors are particularly challenging for the accurate and complete transcription of spontaneous and conversational speech, especially for 9 https://github.com/usnistgov/SCTK social science research where researchers (users) might consider hesitations, false starts and filled pauses important as they convey pragmatic information. Considering impacts of this predictive bias, transcripts of speakers who produce more "fluent" speech are much more easily interpretable. Retrieving speech content and speech style of less fluent speakers as well as some second language speakers, on the other hand, requires more labour and time, potentially negating any benefits of ASR.

Proposed methods
To document predictive bias in ASR in a way that is mindful of 1) user experience, 2) socio-historical and (socio)linguistic context, 3) (potential) harms (re)produced by the system, and 4) technical aspects of ASR, we need to draw on methodologies and knowledge from HCI, sociolinguistics, research on fairness in AI, and SLT.

Intersectional benchmarks
ASR systems are usually evaluated in terms of their WER, for one or more unseen test sets (often including well-established benchmark sets). As seen in the case study above, word error rates vary strongly across individual recordings and speakers, and (benchmark) test sets (e.g. Barker et al., 2017) are becoming increasingly naturalistic and (potentially) diverse; a recent state-of-the-art system by Google (Chiu et al., 2018) was trained and tested on "representative" data drawn from Google's voicesearch traffic. However, even assuming that the test sets are representative of the developer's users, it is 1) not clear that the intended or current user base is reflective of all use cases or potential users (especially if the system is sold to third parties as with GC STT), and 2) possible or even likely that significant variation in performance between user groups is hidden by reporting an average across all tested recordings. Importantly, as Black feminist scholarship has pointed out, multiple demographic axes linked to interlocking structures of oppression (e.g. race and gender) cannot be considered separately (Crenshaw, 1991). It is thus important that in addition to disaggregating by language variety to also consider, for example, gender to create an "intersectional" benchmark (see also Costanza-Chock, 2020). This approach has been successful in highlighting disproportionate predictive bias for particular subgroups in other ML domains (e.g. darkerskinned women in facial analysis: Buolamwini and Gebru, 2018;Raji and Buolamwini, 2019), and SLT (Jiang and Fellbaum, 2020).
To apply an intersectional benchmark to a larger sample of the Lothian Diaries, we intend to match short audio snippets with the same reference transcript produced by different speaker groups to isolate pronunciation effects, and look systematically at potential differences in content and speech style (following Koenecke et al., 2020).

Qualitative error analysis
Intersectional benchmarks alone are not enough however, as WER does not account for the context or effect of an error. Understanding the context of errors is useful since errors are both more likely to occur and to be severe in particular phonetic, prosodic and lexical contexts. Like us (though working with a very different system and data), Goldwater et al. (2010) find that words before or after hesitations, repetitions and word fragments, turn-initial words and infrequent words are more likely to be misrecognised and that erroneous substitutions are often different forms of the same lexeme (e.g. ask/asked). While some of these errors can be easily disambiguated through context, others (e.g. can/can't) could be quite disruptive to communication. Word errors can also lead to domino effects, where one wrongly decoded word feeds into further erroneous predictions (Martin and Tang, 2020). While metrics which are more sensitive to the type and context of the error or directly model human evaluations have been proposed (Nanjo and Kawahara, 2005;Morris et al., 2004;Mishra et al., 2011;Kafle and Huenerfauth, 2020) they are not widely adopted and extensive qualitative error analysis is rare. A context-sensitive approach would be particularly interested in the type of error and its effect given the linguistic context.

User experience
Evaluations of SLT systems rarely reflect explicitly on how users interact with them 10 . However, because both (perceived) severity and impact as well as prevalence of errors depends on recording and task, understanding how people use ASR-based technologies in their daily life is important. Future work concerning predictive bias in ASR would benefit from incorporating HCI methodologies like interviews, ethnography and qualitative surveys to gain a deeper understanding of users' experiences. So far, researchers in HCI have been particularly interested in how people interact with voice user interfaces (e.g. Porcheron et al., 2018;Luger and Sellen, 2016), though little attention has been paid to the role of accent and dialect. Furthermore, especially given the context of the recent shift to increased remote work and education, applications of cloud-based speech recognition for personal or business use extend beyond voice user interfaces to automatic captioning of audio and video lectures and meetings. Domain-general and naturalistic recordings of continuous spontaneous speech pose a particular challenge to ASR systems, and insights into what types of errors users perceive to be particularly disruptive and common depending on their linguistic and demographic background should inform development and evaluation of ASR systems. For example, in the context of the Lothian Diary Project the goal of ASR is to produce transcriptions which can be used by linguists and other social science researchers to analyse both what participants are saying and how they are saying it. Every aspect of their speech, including disfluencies and repetitions as well as specific lexical choices (e.g. past tense vs present tense) are relevant to this analysis and should as such be preserved in a transcript. Furthermore, because most speech in this context is largely unplanned, higher error rates around disfluent or informal speech are particularly disruptive. When applying the proposed methodology to other use cases (e.g. automatic captioning of video lectures or business meetings) interviews with stakeholders can clarify what types of errors are particularly disruptive.

Considering context and impacts
Considering the broader societal context in which an ASR system is developed and implemented allows us to identify the specific harms it could inflict on users and (sometimes at least) see the underlying societal structures giving rise to predictive bias. Identifying risk and causes in turn allows us to mitigate harms (and, in future systems, bias). In the case of commercial ASR (in English), research suggests that predictive bias is a result of under-representation of varieties of marginalised speaker groups in proprietary training and test sets. For many open source and licensed corpora used to train and benchmark ASR systems, incomplete documentation makes it difficult to es-timate representation; the commonly used Switchboard (Godfrey and Holliman, 1993) and TIMIT corpora (Garofolo et al., 1993) (both US English) and Mozilla's recent open-source Common Voice corpus 11 , for example, do not record speaker race. The speaker characteristics of training sets depends on the broader societal context. For example, use of commercial speech recognition (e.g. in the case of Google's system) and participation in scientific studies (e.g. the licensed corpora) or crowd-source tasks (e.g. Mozilla Common Voice) differs across demographic groups (for example based on income and education). Imbalanced corpora are also tied to ideologies around whose ways of speaking are considered "legitimate", "correct" or "native".
Some of the more obvious specific harms of predictive bias include difficulties using voice user interfaces, which for some users are crucial assistive technology. As ASR spreads into high-stakes contexts such as hiring, substantial harms could be incurred if systems perform worse for already marginalised groups, effectively encoding "accentism" and linguistic prejudice in automatic systems. Even assuming no prediction bias across different speaker groups, the use of ASR in automatic analysis of video interviews to recommend or rank applicants (e.g. HireVue 12 ) risks real harm in the case of even small recognition errors and potentially entrenches existing language ideologies around "professional", "fluent" or "competent" speech patterns. For example, HireNet (Hemamou et al., 2019) extracts information about prosody and speech fluency to predict "hireability" (as annotated by recruiters). Other harms include less usable automatic captions and potential downstream effects as described in our case study.

Conclusion
We have proposed an approach to ASR evaluation which considers the experiences of different user/speaker groups, sociolinguistic context and potential impacts of predictive bias. We argue that this interdisciplinary approach is necessary to significantly advance our understanding of ASR usability. We particularly invite perspectives from the fields of human-computer interaction in order evaluate speech and language technologies as systems situated in specific sociolinguistic and socio-technical contexts which perform specific tasks for specific (language) users.