{'fraction_non_alphanumeric': 0.041259377131166175, 'fraction_numerical': 0.019777222095930892, 'mean_word_length': 4.846179401993355, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Integer-only Zero-shot Quantization for Efficient Speech Recognition

End-to-end neural network models achieve improved performance on various automatic speech recognition (ASR) tasks. However, these models perform poorly on edge hardware due to large memory and computation requirements. While quantizing model weights and/or activations to low-precision can be a promising solution, previous research on quantizing ASR models is limited. In particular, the previous approaches use floating-point arithmetic during inference and thus they cannot fully exploit efficient integer processing units. Moreover, they require training and/or validation data during quantization, which may not be available due to security or privacy concerns. To address these limitations, we propose an integer-only, zero-shot quantization scheme for ASR models. In particular, we generate synthetic data whose runtime statistics resemble the real data, and we use it to calibrate models during quantization. We apply our method to quantize QuartzNet, Jasper, and Conformer and show negligible WER degradation as compared to the full-precision baseline models, even without using any data. Moreover, we achieve up to 2.35x speedup on a T4 GPU and 4x compression rate, with a modest WER degradation of<1% with INT8 quantization.

One promising solution to tackle this challenge is to quantize model weights and/or activations to low-precision. This provides multiple benefits. First, quantization reduces the memory footprint by storing weights and/or activations in lowprecision. For instance, INT8 uses 4× less memory relative to FP32. Second, quantization accelerates execution and decreases power consumption by using specialized hardware for low-precision arithmetic [6,7,8]. Following its success on many computer vision [6,7] and natural language processing tasks [8,9,10], there have been attempts to apply quantization to ASR models [11,12,13,14]. However, prior quantization work lacks two important features: integer-only quantization and zero-shot quantization.
Integer-only quantization [6,7,8] is a quantization scheme where all operations (e.g., convolution and matrix multiplication) are performed using low-precision integer arithmetic. To the best of our knowledge, all the prior quantization methods for ASR models use simulated quantization, where all or part of operations are performed with floating point arithmetic. For instance, [13] only performs convolution with integer arithmetic and leaves ReLU and Batch Normalization (BatchNorm) as floating point operations. Compared to simulated quantization, integer-only quantization can decrease latency and power consumption by fully utilizing efficient integer processing units. More importantly, it also allows deployment on popular and highly optimized edge processors designed for embedded, mobile, or IoT devices that often do not support floating point arithmetic. ARM Cortex-M [15], GreenWaves GAP-9 [16], and Google's Edge TPU [17] are some examples of edge processors without dedicated floating point units.
Zero-shot quantization [18,19,20,21,22,23] is a quantization scheme that does not require any training or validation data. This is important because datasets are not always available, especially for ASR use cases such as smart speakers or healthcare, where privacy and security are concerns. However, prior work is based on either quantization-aware training [11,12,13,14] or post-training quantization [13,14], both of which require access to training and/or validation data to finetune or calibrate the quantized models. As such, these methods cannot be applied if no data is available.
We propose to address the aforementioned limitations of the prior quantization work for ASR models. In particular: • We develop an integer-only, zero-shot quantization scheme for ASR models, which allows efficient deployment on integer processing units and achieves negligible WER degradation even without any access to training and/or validation data ( § 2.2 and § 2.3). • We apply our method to QuartzNet-15x5 [5], JasperDR-10x5 [1], and Conformer-Large [2] and evaluate their wordto-error-rate (WER) on the Librispeech benchmark [24]. With 8-bit quantization for weights and activations, we achieve negligible WER degradation of up to 0.29%, 0.08%, and 0.87% for each model, respectively ( § 3.1). • We deploy INT8-only QuartzNet on a T4 GPU, and show up to 2.35× speedup compared to its FP32 counterpart ( § 3.2).

Basic Quantization Method
In this work, we use uniform symmetric quantization. This method uniformly maps a real number x to an integer value q: is the quantization bit-width, Q is the quantization operator, Int is the round operation, clip is the truncation function with the clipping parameter α, and S is the scaling factor defined as α/(2 b−1 − 1). The reverse mapping from the quantized values q to the real values (aka dequantization) isx = DQ(q, S) = Sq ≈ x, where DQ denotes the dequantization operator.
Determining the clipping range [−α, α] that best represents the range of the input x is a primary challenge of quantization. One popular choice in practice is to use the minimum and maximum values of the input x, i.e., α = max(|x max |, |x min |). However, this approach is susceptible to outliers. An unnecessarily large clipping range due to a few outliers increases the rounding error of quantized values within the range. One way to alleviate this is to use percentile values, e.g., 99% smallest/largest, instead of the min/max values.
In dynamic quantization, clipping ranges are computed during inference. However, calculating input statistics (e.g., min, max, and percentile) can be costly in real-time, and requires floating point operations that would prevent us from doing integer-only quantization. Therefore, in this work, we only consider static quantization where we pre-compute the clipping ranges and fix them during inference as in [6,7,8]. It is straightforward to pre-compute the ranges for weights as they have fixed values during inference. However, activations vary across different inputs, and therefore their ranges also vary. One popular method to address this is calibration, which runs a series of training data to compute the typical range of activations. Later in § 2.3, we extend this idea and show how to calibrate without training data. We refer the interested readers to [25] for more details in quantization methods.

Integer-only Quantization
Integer-only quantization [6,7,8] not only represents the model weights and activations with low-precision integer values, but it also carries out the entire inference with integer arithmetic. Broadly speaking, the core of integer-only quantization is the linear property of the operations. For instance, let q W and q x denote the quantized values for weightW and activationx, respectively. That is,W = S W q W andx = S x q x where S W and S x are the scaling factors forW andx ( § 2.1).
due to the linear property. Therefore, we can apply integer convolution directly to the quantized values q W and q x without having to dequantize them intoW andx and apply floating point convolution [6,25]. This property holds true for convolution, matrix multiplication, and ReLU activation, which are the basic building blocks for neural architectures. In addition, BatchNorms are folded into the preceding convolution layers [6], and non-linear activations (e.g., Sigmoid, Softmax and Swish in Conformer) are approximated with 2nd-order polynomials following the procedure in [8] to avoid their floating point execution.

Zero-shot Quantization
Quantization generally requires training data (1) to precompute the clipping ranges for calibration (Equation 1) and/or (2) to finetune the quantized model to recover accuracy degradation from quantization. However, in many cases, pre-trained models are distributed without the data they were trained on due to proprietary, privacy, and security concerns. As such, multiple zero-shot quantization schemes have been proposed, mostly in the field of computer vision, to allow quantization in such cases. A widely adopted method is to generate synthetic image data that can be used in place of the real training data for calibration and/or finetuning [18,19,20,21,22,23].
Similar to [19,21], we generate synthetic data with a similar data distribution to the training data. In particular, we generate the data to match the runtime statistics (i.e., hidden activation distributions) of the training data using the running means and variances stored in BatchNorm layers. Note that the running means and variances capture the runtime statistics of the training data. Let x denote the synthetic data, and µ i and σ i the mean and variance of the activation from the i-th BatchNorm with x as an input. Additionally, letμ i andσ i denote the running mean and variance of the BatchNorm. Then, we aim to minimize the loss defined by the Kullback-Leibler (KL) divergence between N (μ i ,σ i ) and N (µ i , σ i ), assuming that the hidden activations follow the Gaussian distribution: In computer vision tasks, input x is often regarded as a learnable parameter and is trained with standard backpropagation to reduce the defined loss [19,20]. However, it is not straightforward to apply this directly to speech data, as we are dealing with time varying audio signals rather than static images. As such, we instead propose to generate synthetic mel spectograms. Mel spectrogram is a widely used preprocessed representation of audio signals [1,2,3,5,26]. Also, its data representation is more suitable for synthetic data generation as  Fig. 1: End-to-end process of synthetic data (i.e., mel spectrogram) generation for zero-shot quantization. Input data is trained with standard backpropagation in a direction that matches the hidden activation statistics of the real data.
it can be regarded as a 1-dimensional image. Specifically, our zero-shot quantization scheme consists of two steps.
• Synthetic data generation: A batch of 1-dimensional arrays is randomly initialized and set as a learnable parameter. It is then fed into the full-precision (i.e., non-quantized) model to compute the loss in Equation 2. This minimizes the KL divergence between two distributions, one from the real training data and the other from the synthetic data, for each BatchNorm activation. Using backpropagation, the input is iteratively trained using gradient descent to minimize the loss. Figure 1 illustrates the end-to-end process. • Calibration: The synthetic data is fed into the target model in place of the real mel spectrograms to determine the clipping ranges for all activations. Then the model is quantized according to Equation 1.
The benefit of this approach is that we can use this synthetic data to perform quantization, and avoid the need to get direct access to the training data which may not be available.

RESULTS
In this section, we first demonstrate in § 3.1 that the quantized models can match the accuracy of full-precision baselines. We then evaluate the latency speedup of the quantized models on real hardware in § 3.2. As baseline models, we select QuartzNet15x5 [5] and JasperDR-10x5 [1] as CNN-based architectures, and Conformer-Large [2] as a more complex architecture with multi-head attentions and non-linear activations such as Sigmoid, Softmax, and Swish. Such non-linear activations can be computed with integer arithmetic by approximating them with 2nd-order polynomials as described in [8].
We note that our quantization method is not a model-specific solution, and can be applied to many other ASR models.

Accuracy Evaluation
For accuracy evaluation, we use the NeMo [27] implementation of the pre-trained QuartzNet-15x5, JasperDR-10x5, and Conformer-Large for the full-precision baseline models. The synthetic data is initialized with the uniform distribution from [−0.3, 0.3], and is trained using the Adam optimizer [28] with batch size 8, β 1 = 0.9, β 2 = 0.999, and learning rate of {0.03, 0.04, 0.05, 0.06} for {200, 250} iterations. We generate 20 such batches and use them for calibration. Figure 3 contains some examples of the generated synthetic data. For calibration, we use the min/max values for clipping. We quantize the baseline models with W8A8 (i.e., 8-bit weights and activations) and W6A8, and do not finetune the models after quantization. All the reported numbers are averaged over 4 different runs (i.e., synthetic data generation and calibration). Both the baseline and quantized models are evaluated on the widely used LibriSpeech [24] benchmark. Table 1 compares the WER of the baseline and quantized models with different bit-width settings on dev-clean/other and testclean/other datasets, where hyperparameters are tuned on the dev sets. We also include the model size and BOPs (bitoperations) of each model for comparison. BOP is the total number of bit operations and is a hardware-agnostic proxy to model complexity [29]. For QuartzNet, the W8A8 setting results in a modest WER degradation of 0.23% and 0.29% on the test-clean/other datasets with 4× reduction of the model size as compared to the full-precision baseline. The W6A8 setting further reduces the model size to ∼5× of the baseline only within 0.50% and 0.97% WER degradation on the test sets. On Jasper, the quantized model exhibits negligible WER increase of up to 0.08% and 0.19% on the test sets with the W8A8 and W8A6 settings, respectively. Despite the complexity of the Conformer architecture, INT8-only version only exhibits 0.87% WER degradation. Note that all the reported values are without any finetuning after quantization.

Latency Evaluation
We evaluate the latency speedup of quantized models by direct deployment of the quantized QuartzNet on a Tesla T4 GPU with the Turing Tensor Cores that support accelerated INT8 execution. We use NVIDIA's TensorRT library [30] for model deployment. All the experiments were conducted on Google Cloud Platform virtual machine with a single Tesla T4 GPU, CUDA 11.1, cuDNN 8.0, and TensorRT 7.2.1. Although we select T4 GPU as our target device due to its extensive software support [30], we should highlight that our approach is not specific to GPUs, and one could also deploy the quantized models to other processors. For evaluation, we use QuartzNet-5x5, 10x5, and 15x5, which are the small, medium, and large variants of the QuartzNet family [5]. We measure the inference latency using a 10-second audio input with a sampling rate 16K, and test the W8A8 (i.e., INT8-only) setting for the quantized model. Figure 2 compares the latency and model size of the baseline and quantized QuartzNet models with different size configurations. As shown in the plot, INT8-only QuartzNet15x5 achieves 2.35× speedup and 4× compression rate as compared to the FP32 baseline. Also, the quantized QuartzNet15x5 matches the WER of the largest model, while only needing the computation and memory requirements of the smallest model. This is an important observation for practitioners who wants to design an efficient model, as the alternative approach of using a full precision but shallower model can lead to a sub-optimal solution compared to quantization.

Ablation studies
Here, we show the effectiveness of the zero-shot quantization scheme. To do so, we quantize QuartzNet15x5 using a set of random calibration data from the uniform distribution in [−3, 3]. We observe significant WER degradation when we calibrate the model with random data instead of the synthetic data of our zero-shot quantization scheme. With random data, the test-other WER degrades by 2.47% and 4.02% for W8A8 and W6A8, respectively, which are noticeably higher than the corresponding values with the synthetic data as shown in Table 2. The results confirm the need to generate synthetic data that well represents the actual data used in training.

Analysis of Synthetic Data Generation
In this section, we briefly analyze the synthetic data, which we start with a randomly initialized array (second row of Figure 3) and solve Equation 2 to generate synthetic mel spectrograms. We show two such examples in the last three rows of Figure 3. First, note that the synthetic data can capture the local patterns  Fig. 3: Comparison of the real (1st), random (2nd), and synthetic (3rd-5th) mel spectrograms. Below each synthetic mel spectrogram is the text that it decodes to.
observed in real mel spectrograms. Second, we observe that if the synthetic data is fed into QuartzNet followed by a 3-gram language model, it can be decoded into meaningful words and sentences, instead of a sequence of random characters. Examples of the decoded sentences are captioned below the corresponding synthetic data in Figure 3. This is a further evidence that the synthetic data is close to the actual training data, and it explains why it enables good quantization results.

CONCLUSION
In this work, we propose a zero-shot quantization scheme for ASR models that only uses integer-only computation. Our proposed method entirely removes floating point operations from inference, thereby allowing efficient model deployment on fast and power-efficient integer processing units. In addition, it does not require any training and/or validation data, as it calibrates models with synthetic data that resembles the real data. Our quantization scheme exhibits a large compression rate of 4× with small WER degradation of 0.29%, 0.08%, and 0.87% on QuartzNet, Jasper, and Conformer for INT8only quantization. Furthermore, we achieve 2.35× speedup by directly deploying INT8-only QuartzNet on a T4 GPU.