{'fraction_non_alphanumeric': 0.03173216885007278, 'fraction_numerical': 0.010131004366812227, 'mean_word_length': 4.974260869565217, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Toward Zero Oracle Word Error Rate on the Switchboard Benchmark

The"Switchboard benchmark"is a very well-known test set in automatic speech recognition (ASR) research, establishing record-setting performance for systems that claim human-level transcription accuracy. This work highlights lesser-known practical considerations of this evaluation, demonstrating major improvements in word error rate (WER) by correcting the reference transcriptions and deviating from the official scoring methodology. In this more detailed and reproducible scheme, even commercial ASR systems can score below 5% WER and the established record for a research system is lowered to 2.3%. An alternative metric of transcript precision is proposed, which does not penalize deletions and appears to be more discriminating for human vs. machine performance. While commercial ASR systems are still below this threshold, a research system is shown to clearly surpass the accuracy of commercial human speech recognition. This work also explores using standardized scoring tools to compute oracle WER by selecting the best among a list of alternatives. A phrase alternatives representation is compared to utterance-level N-best lists and word-level data structures; using dense lattices and adding out-of-vocabulary words, this achieves an oracle WER of 0.18%.


Introduction
This work is about the very well-known "Switchboard" subset of an evaluation of US English conversational telephone speech recognition, originally conducted by NIST in 2000. [1] The current best published result is 4.3% WER [2], which also acknowledged that "most of the speakers appear in the training data, hyperparameters are optimized on [the test set], and the human error rate might also have been overestimated".
Other recent results demonstrate 5.0% with low-latency streaming [3], and many works reference [4] and [5] as the first systems to achieve the milestone of parity with human performance, which is described as 5.1% to 5.9% WER.
A careful analysis in [6] notes that "humans are more likely to miss words than to misrecognize them", and is notable in several regards: code was provided to specify a non-standard data cleaning and text normalization process, while output from a research system was re-scored in an (unsuccessful) attempt to replicate a published result. Our work continues in this effort to fully describe and improve upon the standard scoring methodology, sharing data and software to enable reproducible results.
This work benchmarks commercial ASR systems, inspired by [7], which archived outputs from the dates of collection. For this Switchboard benchmark, a particular advantage of benchmarking commercial systems is that the evaluation simulates a more realistic scenario of presenting each conversation side as an entire 5-minute audio file. By contrast, the NIST evaluation allowed research systems to use the reference segmentation as input, which can result in artificially low WER scores.
This work is similar to [8] by presenting transcript precision and recall as possibly more insightful alternatives to WER, particularly for highlighting characteristics of human performance. The use of an "oracle" word error rate that is optimistically calculated from ASR alternatives is similar to [9] which reranks N-best alternatives for spoken content retrieval, as well as our prior work [10] in evaluating systems for spoken term detection.
While evaluating traditional N-best lists, we also introduce a novel representation for phrase-level alternatives. This captures the full expressiveness of an ASR lattice [11], but in a more compact and linear data structure that can be conveniently manipulated as input to ASR scoring software, or indexed by a text-based search engine infrastructure. The aim of this work is to show how this representation enables nearly perfect oracle accuracy (0.18% WER) on a well-established ASR task. This theoretical result motivates the further use of phrase alternatives toward a highly practical goal of enabling spoken term detection (i.e. audio search) applications that exhibit perfect recall.

Corrected reference files
Reference files from the original NIST evaluation are now distributed by the Linguistic Data Consortium (LDC) 1 , but differ from what was later used in DARPA-funded evaluations known as "RT-03" and "RT-04F". For example, the newer GLM files include backchannel mappings that generally improve scores.
Human transcribers disagree on this very difficult task [4,5,6,12,13], so it should not be surprising that there are inevitably some errors in these reference transcripts and mappings. For this work, a professional linguist was commissioned to very carefully audit and correct these references. In addition to the original transcripts, they could refer to four independent results from human speech recognition (HSR) services, but not any of the ASR systems. This paper's authors further corrected the GLM file with ad-hoc normalization of number formatting.
However, the vast majority of changes were related to an artifact of the make_reference script that is distributed with the test set; it was used to create the reference STM by converting transcripts from an original TXT file format. Unfortunately, every contraction in the original transcript is always expanded into multiple words (see lines 126-131 in make_reference). This does not seem to be sensible, especially considering that the GLM filtering would also redundantly expand all contractions. We thus decided to reverse this automatic expansion of contractions, and directed the highly skilled linguist to transcribe each instance correctly as either its contracted or expanded form by carefully listening to each acoustic realization, favoring the contracted form in cases of true ambiguity.
These corrected reference files are shared publicly, 2 and should lead to substantial improvements across all systems.

Expansions vs. alternations
The NIST SCTK software 3 can use a GLM mapping file to filter reference STM and hypothesis CTM files by applying a set of transformation rules. For example, contracted or compound words can be expanded with a rule such as I'M => I AM.
However, by always expanding contractions in both the reference STM and hypothesis CTM, this rule often doublecounts correct matches as well as errors. A better approach is to denote alternations to be applied in the GLM file, e.g. I'M => { I'M / I AM }, which will be scored as one or two matches or errors as appropriate. The original form should be listed first in the alternation, since SCTK will favor it when multiple alignments have the same number of errors; otherwise, favoring the expanded form results in overly optimistic scoring.

Optional deletions and excluded words
Another effect of the filtering is to treat some words as optional deletions, marked by parentheses in the STM file, in particular (%HESITATION). An ASR system should exclude such difficult words from CTM hypotheses, due to the asymmetric risk: an error can have a larger effect on the numerator of WER, compared to a correct match incrementing the denominator (Eq. 1). One major commercial system (ASR3) never hypothesizes hesitations -nor any backchannels such as "uh-huh", which are not optional deletions under the NIST scoring rules. So that their system is not disadvantaged by a design choice, we can consider backchannels to be optional deletions as well. So that other ASR systems are not then disadvantaged by hypothesizing backchannels, we also exclude those from their CTM files.

Segmentation
The NIST tools can misalign hypotheses with word-level timestamps that differ slightly from the reference utterance-level segmentation of an STM file. A solution is to convert the multisegment STM into one long segment. This can improve WER for ASR systems with consistent timestamp drift, and is needed to score any HSR (human speech recognition) result.
This problem is not observed in academic research experiments, because the reference segmentation is assumed to be a valid input to the ASR system. This practice may be unrealistic in real-world settings, however, as seen in the bottom rows of Whereas the WER metric can be computed as in Eq. 1, a pair of non-standard metrics can also be useful to consider when evaluating ASR accuracy. Transcript precision is the proportion of hypothesized words that are correct. It does not penalize deletions and scores consistently well for human transcripts, since it forgives the common tendency to omit words or phrases that do not convey much meaning (e.g. stuttering "i i i i ..."). The recall metric can be rather variable for HSR results; it could be useful for evaluating against non-verbatim reference transcriptions.

Representations of ASR Alternatives
Lattices can be generated by some ASR decoders, particularly in a WFST system such as Kaldi [11], to represent the inherent ambiguity and uncertainty of hypotheses. However, the lattices are large and difficult to use in applications that require properties such as time-synchronous word sub-sequences. Let Lu be the formal language representing the set of all word sequences encoded in the lattice for a given utterance u.

Utterance-level alternatives (i.e. N-best lists)
Utterance-level alternatives, better known as N-best lists, can be used to enumerate a formal language Lu(N ), a set comprising up to N most likely word sequences in the lattice. The lattice's language is a superset, with equality in the theoretical limit:

Word-level alternatives
Word-level alternatives, sometimes known as sausages, can be derived by aligning paths in a lattice [14] or from statistics used in Minimum Bayes' Risk decoding [15]. These represent a smaller formal language of up to N single-word sequences Lw(N ) at each word position w. Due to 1-to-1 word alignments, the lattice's language cannot be decomposed as a crossproduct and concatenation (indicated by ) of component sets: There may be sequences in Lu that cannot be represented as a concatenation of elements in Lw(N ), even for large N .

Phrase-level alternatives
By contrast, all paths in the lattice can be represented as a subset of the crossed and concatenated phrase-level alternatives [16]: In this formulation Lp(N ) is a set of up to N word sequences, which may be of varying lengths, at phrase position p.

Converting lattices to phrase alternatives
Phrase alternatives can be derived from a lattice as follows: 1. Word-align the lattice, which may need determinization.
2. Establish phrase boundaries as those times not crossed by non-silence arcs (above some arc posterior threshold).
3. For each phrase, mask the lattice arcs outside the phrase boundaries by setting their output symbols as epsilon.
The phrase alternatives representation is motivated by its compactness compared to utterance-level alternatives, since it decomposes the utterance as a concatenation of word sequences that are assumed to be independent of each other. It is also more expressive since this cross product generates additional word sequences that may not have been present in the lattice.

Representing alternative hypotheses in NIST SCTK
A lesser-known feature of the CTM file format is that it can be used to represent alternatives in ASR hypotheses, for example: sw_4390 A * * <ALT_BEGIN> sw_4390 A 4.49 0.66 UM sw_4390 A * * <ALT> sw_4390 A 4.49 0.66 I'M sw_4390 A * * <ALT_END> While this is typically used to represent alternations created by filtering with the GLM file, it can be further leveraged to enable oracle scoring of ASR alternatives at various levels. However, this functionality requires a minor modification 4 to the sclite source code, as well as auxiliary software 5 that can create the CTM files while fixing a couple of related bugs in SCTK (such as expanding doubly-nested alternatives after GLM filtering).

Speech Recognition Systems
Automatic (ASR) and human (HSR) systems were evaluated: ASR1 is a Kaldi baseline. An OPGRU acoustic model and a trigram language model were trained only on Switchboard plus Fisher. These models were loaded by the Mod9 ASR Engine to produce utterance-, word-, and phrase-level alternatives.
ASR1 * customized the decoding graph by adding the 28 words that were out-of-vocabulary (OOV) with respect to the system's relatively small lexicon (about 40,000 words that appeared in the training data). Pronunciations were automatically generated with a grapheme-to-phoneme model [17] by requesting the Mod9 ASR Engine's add-words command.
ASR1 † used non-default pruning beam sizes to produce denser lattices, by requesting a speed:3 option of the Mod9 ASR Engine, a trade-off with more compute and memory usage. ASR1 * † combined both of the above settings.
ASR2 is IBM Watson with an older "Narrowband" model, instead of using a more accurate "next-generation" model, because this system is uniquely capable of demonstrating utterance-and word-level alternatives at extreme depths.
ASR3 is Google Cloud Platform's STT service, using an "enhanced" variant of their "phone call" model.
ASR4 is Amazon Transcribe, configured for US English.
ASR5 is Microsoft Azure's Speech-to-Text service, which generates utterance-level alternatives of very limited depth. ASR6 is the system in [2], from which IBM Research shared CTM-formatted system outputs for evaluation purposes.
HSR1 is the Rev.com service, which has speaker labeling. HSR2 is the TranscribeMe service, requesting "verbatim" quality transcripts that include speaker labeling.
HSR3 is the TranscribeMe service, requesting "first draft" quality transcripts that do not include speaker labeling.
HSR4 is the cielo24 service, with no speaker labeling.

Results
All results can be reproduced from system outputs 6 that were archived in early 2022, using open-source scoring scripts. 7 The bottom row and right column of Table 1, middle section of Table 2, and left columns of other tables have italicized font. This convention is used to clarify which results might be considered unrealistic, due to use of a reference segmentation or also because of the oracle nature of selecting a best alternative. Table 1 presents the WER results from scoring each of the ASR systems with successively improved configurations of the scoring tools, as described in Sections 2.1 through 2.4. Table 2 compares the ASR and HSR systems, including precision and recall metrics in addition to WER. The results for HSR3 and HSR4 are exceptional because they required conversion of reference STM files into a single-channel format, using forced-alignment with an HTK-based ASR system; regions of overlapped speech may be incorrectly merged in some cases. Dual-channel audio files were submitted to the HSR services, so transcribers could understand conversations sides in context. Table 2 also reports the cost of processing the Switchboard test set, based on its duration of 100 minutes. For ASR without reference segmentation, audio was presented as channel-  Tables 3, 4, 5, and 6 report the oracle WER when the NIST SCTK scoring software is presented with CTM files that represent utterance-, word-, and phrase-level alternatives. These results all use the reference segmentation, since the software cannot score alternatives that cross STM segment boundaries. Each table reports the parameter N that was requested, which may be greater than the actual Nmax returned. The N.9 and N.5 columns indicate the depths of alternatives at the top decile and median results; these convey the distribution more clearly than the mean statistic. The rightmost columns report the storage size of the gzip-compressed CTM files in megabytes.
The last row of Table 6 relates a hypothetical oracle selecting the best transcript from a phrase-level representation of alternatives, derived from very dense lattices, decoded with added knowledge of all OOV words, using a reference segmentation.

Conclusion
This work highlighted subtle issues with evaluating the famous Switchboard benchmark. It presented a reproducible Kaldi ASR baseline, comparing major cloud platforms to human transcription services, and clarified that IBM's research system achieves a super-human record of 2.3% instead of 4.3% WER.
Some experiments are unrealistic to varying degrees, ranging from the assumption of an oracle to the accepted use of a reference segmentation. Nonetheless, such results demonstrate the potential for lattice-based ASR approaching 0.18% WER.
These results motivate future work to improve lattice generation [18,19], particularly in E2E ASR systems. Our current research also explores open-vocabulary decoding in a WFST framework, in which novel words may be included in a lattice and derived phrase alternatives. These advances enable new applications, e.g. audio search or machine-assisted transcription, that can be designed to mitigate inevitable errors in 1-best ASR.

Acknowledgments
Thanks to our many friends from ICSI: ⋆ Michael Ellsworth, who carefully audited the references. ⋆ Andreas Stolcke, who clarified many evaluation practices. ⋆ Brian Kingsbury, who shared results from IBM Research. ⋆ Deanna Gelbart, who wrote code for phrase alternatives.