{'fraction_non_alphanumeric': 0.02579117047174156, 'fraction_numerical': 0.011678399894729916, 'mean_word_length': 5.364949748743719, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Confounding Tradeoffs for Neural Network Quantization

Many neural network quantization techniques have been developed to decrease the computational and memory footprint of deep learning. However, these methods are evaluated subject to confounding tradeoffs that may affect inference acceleration or resource complexity in exchange for higher accuracy. In this work, we articulate a variety of tradeoffs whose impact is often overlooked and empirically analyze their impact on uniform and mixed-precision post-training quantization, finding that these confounding tradeoffs may have a larger impact on quantized network accuracy than the actual quantization methods themselves. Because these tradeoffs constrain the attainable hardware acceleration for different use-cases, we encourage researchers to explicitly report these design choices through the structure of"quantization cards."We expect quantization cards to help researchers compare methods more effectively and engineers determine the applicability of quantization techniques for their hardware.


Introduction
The recent success of deep learning has come at the expense of larger models, which have greater computational cost and memory footprint, making the deployment of neural network inference challenging. A common approach for improving the performance of neural network inference is quantization, which represents floating point values in neural networks as low-bit integers [19,22]. By utilizing hardware support for low precision arithmetic, quantization enables higher throughput, decreased memory traffic, and lower storage requirements. * Work completed while the author was with Luminous Computing.
Many techniques have been developed to enable quantized inference in a variety of settings. If adequate resources are available, models can be retrained for multiple epochs while simulating the effects of quantization [22,14], but in other cases, no data may be available for re-training [7,38,31]. Depending on hardware support and available resources, quantization strategies use mixed precision [34,13,12,35,37,3], optimize for more effective quantization grids and clipping ranges [3,9,44], or define mappings from weights to quantized values that are more effective than rounding [36,30,18].
A key challenge in the quantization literature is attributing improvements in accuracy to methods because different methods are evaluated subject to confounding tradeoffs. These confounding tradeoffs represent design decisions that trade off inference acceleration or resource complexity for improved accuracy. In Section 2, we articulate a variety of these confounding tradeoffs, including the quantization method (signed vs. unsigned, per-tensor vs. per-channel), which operations are quantized (e.g. residual connections), mixed-precision constraints, the amount of data and compute required by the method, and the baseline accuracy of the floating point model.
To address this problem, we propose that novel quantization research discuss these tradeoffs through "quantization cards," described in Section 3. Quantization cards are a structure for reporting design decisions, analogous to datasheets for hardware, and more recently, datasheets for datasets [15] or model cards for machine learning models [28]. Such a structure is important for reporting quantization design decisions because unlike other hyperparameters, these design decisions affect the attainable inference acceleration. We propose quantization cards instead of establishing a benchmark because the set of tradeoffs is rapidly growing, such as using integer-only operations excluding division [40] or generating synthetic data [7,38].
We expect quantization cards to be useful for two groups of people. First, researchers of quantization techniques will be able to more accurately compare across methods and replicate prior results. Second, engineers deploying quantized models will be able to assess quantization cards, determine whether the tradeoffs made by the work are applicable for their hardware and application setting, and adopt or combine strategies accordingly. Without quantization cards, these choices are often scattered throughout different parts of the text or even overlooked, making it challenging for users to determine how the method can be used.
We evaluate the impacts of these tradeoffs on quantized model accuracy in Section 4 for uniform quantization and post-training mixed-precision quantization. These experiments demonstrate that changes in confounding tradeoffs can affect accuracy at low bit precision by multiple percentage points, emphasizing the need for quantization cards and unified reporting of design decisions.

Confounding Tradeoffs
Quantization methods are evaluated under a wide array of design choices, each of which trades off inference acceleration or resource complexity. Here, we discuss how those tradeoffs manifest and their impacts on accuracy.
We specifically discuss tradeoffs for uniform quantization, the most common method for quantizing floating point values to low precision [19]. In uniform quantization, floating point values x in some range [x min , x max ] are uniformly mapped to B bit integers by scaling, translating, and rounding the values. Mathematically, this is , and the bias b depends on whether the integers are signed or unsigned. The quantization parameters can be determined by calibration of the range on some subset of data [19] or other strategies such as minimizing the mean squared quantization error [9]. On the other hand, there are various approaches to non-uniform quantization, which may or may not be compatible with bit-operations in hardware [42,29]. We restrict our focus in this paper to confounding tradeoffs for the more commonly deployed uniform quantization.

Quantization Method
Symmetric vs. Asymmetric Quantization Uniform quantization is typically applied via symmetric quantization to signed integers or asymmetric quantization to unsigned integers [22]. Symmetric quantization enforces that x min = −x max , so the bias of the quantization operation is zero, and the floating point value zero is mapped to the quantized integer zero. On the other hand, asymmetric quantization applies to an arbitrary data range, so it can more effectively capture x that is not symmetrically distributed about zero, but consequently will have nonzero bias b. The ability to capture an asymmetric data range is especially important for activations after the ReLU nonlinearity, which discards the sign bit for symmetric quantization. As a result, asymmetric quantizers are expected to outperform symmetric quantizers in accuracy.
However, asymmetric quantization may decrease inference acceleration. In order to evaluate a real valued matrix multiplication using quantized inputs, it is necessary to revert the result by multiplying the bias of x by the sum of the weights in the dot product, and the bias of W by the sum of the inputs [22]. While these sums can be computed at lower complexity than the matrix multiplication, they present a nontrivial and potentially undesirable overhead. Because the sum of x cannot be computed ahead of time, asymmetric quantization of weights presents the larger overhead.
Per-channel vs. Per-Tensor Quantization A frequently discussed tradeoff is between using per-tensor quantization, which quantizes all values in a matrix using the same minimum and maximum, and per-channel quantization, which uses a different range for each channel of a weight matrix or activation tensor [36,30]. Using per-channel quantization improves accuracy at the expense of greater complexity from applying requantization to each channel independently. Moreover, it may be challenging to achieve inference acceleration for dot products if different activations in the same dot product are quantized to different scales.

Quantized Operations Elementwise Operations and Residual Connections
While most quantization methods guarantee that the inputs to every matrix multiplication are quantized, many take different approaches for the quantization of other elementwise operations, such as residual connections. The most common method, and the one implemented in TensorFlow and PyTorch, assumes that the results of every operation are requantized, implying that the inputs to a residual addition are low bit integers, and the output of the elementwise addition is requantized [22]. However, because elementwise operations have a relatively smaller computational complexity than matrix multiplication, they can be executed at higher precision before being requantized [26,8]. Different methods for quantizing residual connections are illustrated in Figure 1. When skip connections are evaluated at high precision, this creates high precision activation propagation through the entire network [33], improving accuracy when using low precision activations.
Full precision elementwise operations, while tractable in many scenarios, may lead to inefficiencies for certain hardware architectures. When memory pressure and bandwidth is a constraint, high precision intermediate feature maps may not fit in cache, and thus must be stored and fetched from main memory. The increased memory pressure may increase inference latency.
Other elementwise operations present similar confounding tradeoffs. For computer vision models, average pooling may be performed at high precision before being requantized, or may be performed directly with low precision integer arithmetic, which leads to information loss [40]. Quantization of average pooling may be overlooked when quantization is simulated with floating point arithmetic. On the other hand, using integer-only arithmetic avoids the cost of casting integers to and from floating point values or performing expensive division for elementwise operations, and improves performance [40]. For natural language processing, transformer models such as BERT contain many elementwise operations, including adding token embeddings to positional embeddings, intermediate nonlinearities such as the softmax, and normalization layers such as Layer-Norm that cannot be fused. These operations are often computed in FP32 [41,43,2], but can be computed more efficiently via integer-only approximations [20].
Excluding First and Last Layer Many works quantize the first and last layer of computer vision models to higher precision, e.g. 8 bits, because they are especially precision sensitive [3]. However, because the last layer of Resnet50 is extremely large, at 3 bits, quantizing the first and last layer at 8 bit precision requires 13% more memory capacity. This difference in compression ratio may improve accuracy at ultra-low precision, so the compression ratio must account for the size of the first and last layer.

Mixed Precision
Different layers of neural networks are tolerant to different degrees of precision, and uniformly quantizing all layers of a neural network to the same low precision leads to accuracy degradation [12]. Mixed-precision quantization uses varying precision for different layers [34,13,12,35,24,37,32] or channels [3,39] of neural networks to decrease the average precision required by neural networks.
Mixed precision allows for different settings for the number of bits. For example, some works restrict the bitwidth for quantized operations to {2, 4, 8} bits [7], whereas other works allow any integer bitwidth [34,3]. While arithmetic on 3-bit integers may not be directly supported in hardware, and operations must be computed by casting these integers to 4-bits, such approaches can save memory traffic if values are efficiently packed. Describing the choice of bitwidth is critical because one mixed-precision method may outperform another solely because it has a larger selection of bit precision settings.
Moreover, the activation bitwidth utilized by a mixedprecision network may be constrained by different desiderata. If memory capacity is the limiting constraint for acceleration, then activation bitwidths can be constrained solely by the maximum feature map size [34]. However, if memory pressure or computational throughput is the system bottleneck, mixed precision methods may be constrained by the average feature map size. The choice of constraint will impact accuracy.

Resource Complexity
Most approaches to quantization are split into three categories, zero-shot quantization, post-training quantization, and quantization-aware training, each of which require different amounts of data and computation. Post-training and zero-shot quantization are typically applied in scenarios where access to data may be limited due to e.g. privacy concerns [7,3]. However, we observe substantial variation within each of these categories in their data and computational complexity.

Data Complexity
The data complexity of different methods may be viewed on a sliding scale. Approaches for posttraining quantization assume different amounts and types of data, ranging from 256 [25] to 1024 examples [30], and sometimes using one example per class [18]. In addition, some methods only require unlabeled calibration data, whereas others (such as the one presented in this paper) require access to labeled data, which may be prohibitive in privacy constrained settings. However, it is possible to generate synthetic labeled data from pretrained models and to use this data for quantization [7,38]. This may lift the stringent <1000 sample requirement for post-training quantization, and instead enable post-training quantization with the only requirement that it be performed rapidly.
Computational Complexity Similarly, the computational complexity of different methods ought to be viewed on a sliding scale. There is a growing body of work that uses different types of optimization or even backpropagation for post-training quantization, but that can complete within 10 minutes on a GPU [30]. Moreover, methods that perform quantization-aware training for a single epoch versus fifty epochs are different in their applicability; the for-

Pretrained Model Accuracy
When evaluating post-training quantization for Resnet50, different works use floating point baseline models ranging in validation accuracy from the torchvision model with 76% accuracy [3,9,25] to the pytorchcv model with 77.7% accuracy [7,18]. However, most works compare their results to those reported by previous papers, so post-training quantization works that use a model with 1.7% better floating point accuracy will by default outperform prior work when the accuracy degradation is small. Thus, studies should note baseline accuracy and relative accuracy degradation when comparing across works. They should also default to baselines with the highest accuracy, as suggested in [40].

Quantization Cards
We propose that researchers of quantization methods describe the design tradeoffs used when evaluating their method through "quantization cards." Quantization cards are broken down into the five categories from Section 2 and explain the tradeoffs made for each of these categories. The categories and specific tradeoffs within each category will grow over time as new tradeoffs are developed. In Table 1 we show the generic questions that should be answered by a quantization card, and in Section 5, we provide an illustrative example. When multiple design settings are used for different experiments in a single work, a single quantization card for the strongest method in the paper with comments on deviations from those settings is sufficient.
Quantization cards may be useful to both researchers of quantization methods and engineers deploying quantized models. First, quantization cards identify often overlooked tradeoffs that impact accuracy by >1%, which can be larger than the difference between state-of-the-art techniques. This helps distinguish improvements of novel methods from those of design decisions and facilitates better comparisons across methods. Second, they improve reproducibility by guaranteeing that all confounding tradeoffs are described. Third, by reporting these tradeoffs in a single location, engineers can at a glance determine whether these tradeoffs can be made by their system, or whether the technique must be adapted. Engineers require this information to be clearly presented because it affects the attainable inference acceleration and the use cases in which methods can be deployed, unlike other hyperparameters, which only affect reproducibility of results.
We suggest the use of quantization cards instead of creating a new benchmark because the set of design tradeoffs is rapidly evolving. For example, some methods restrict operations to integer-only arithmetic excluding division [40], approximate nonlinear operations such as softmax using integer-only arithmetic for NLP applications [20], or generate synthetic data, which may alleviate stringent data constraints for post-training quantization [38,7]. Because these tradeoffs involve novel ways of implementing quantization, they cannot be captured by a fixed benchmark. Instead, quantization cards are modeled after proposals for describing characteristics and use cases of models [28] or datasets [15,17,4,1] to address issues related to fairness and bias in machine learning. They also resemble guidelines for transparent reporting of prediction models in health [10].

Method
We design experiments for each of the design tradeoffs in Section 2 and evaluate results on Resnet50 [16] on the Imagenet dataset [11]. Experiments assume uniform precision unless otherwise noted, keeping the first and last layer at 8 bit precision. For experiments using uniform quantization at low activation bitwidth, we use percentile clipping of activations at the 99.99th percentile [23,27].
For experiments on mixed-precision quantization, we propose a method to stress test the minimum attainable bit precision of different computations. We do so by leveraging the method in [34] to learn the optimal bitwidth for weights and activations in each layer for a pretrained model. This amounts to solving an optimization problem over the bitwidths of each layer to minimize the task loss with an added penalty for the average bitwidth of weights and activations. Experiments using learned mixed-precision quantization are repeated with 5 random seeds, and mean and standard deviation of results is plotted. Uniform quantization has lower variance because it does not use learned parameters, so we evaluate results with a single seed. The

Results
Observation 1: Asymmetric quantizers require fewer bits than symmetric quantizers for equivalent accuracy.
We test the impact of asymmetric quantizers on activation quantization, where we expect that applying symmetric quantization to unsigned activations will require one additional bit due to the discarded sign bit. We quantize all operations except for residual connections in Resnet50, which corresponds to quantizing all operations after the output of a ReLU nonlinearity. We report results in Figure 2. Accuracy does not precisely match up between 4 asymmetric bits and 5 symmetric bits because both must include zero in their representation.
The impact of asymmetric quantizers on weight quantization is smaller since weights are typically signed, although their distributions may be skewed. The number of quantization bins necessary to represent weights with equal fidelity for symmetric and asymmetric quantizers can be measured by the ratio of their ranges, i.e. W max − W min versus 2 (max (W max , −W min )). If the number of bins could be modulated (as opposed to the coarser-grained number of bits), using asymmetric weights would require on average 0.23 fewer bits for Resnet50, or approximately 730 KB. To test the impacts of asymmetric weight quantization when the bitwidths must be discrete, we evaluate mixed-precision weight quantization for asymmetric quantizers and symmetric quantizers that use an additional 0.23 bits on average, reported in Figure 3. At low bitwidths, asymmetric quantization improves accuracy by multiple percentage points.    Observation 2: Per-channel quantization outperforms per-tensor quantization.
We confirm this well-known observation for weight quantizers in Figure 4. We compare the three methods of quantizing residual and skip connections from Figure 1 on their accuracy at low activation precision. Results in Figure 5 demonstrate that methods such as [36] that attain extremely high accuracy with 3 bit activations may be able to do so in part because they do not quantize residual and skip connections.  mixed-precision bitwidth allocations that quantize to an average of 4 bits for weights and activations over all layers and compare against mixed-precision models that keep the first and last layer at 8 bits while quantizing all other layers to an average of 4 bits. While the difference in compression might appear small, results in Table 2 show that it can affect accuracy by almost 1%.
Observation 5: The set of allowed bitwidths for mixedprecision quantization affects the ability of mixed precision to increase accuracy. We test the impact on performance of using any integer bitwidth versus the subset {2, 4, 8}. We implement a restricted version of the post-training quantization method that rounds the continuously parameterized bitwidth to the closest of the three discrete values {2, 4, 8} during training. As shown in Figure 6, restricting the set of bitwidths used by the mixed-precision method does not affect accuracy at high average bitwidths, while at lower bitwidths the degradation is large.
Observation 6: Mixed-precision activations constrained by the maximum feature map size have higher accuracy than uniform precision with the same maximum feature map size.  We compare two settings: mixed-precision activations with a maximum feature map size, and uniform precision activations with the same maximum feature map size. For mixed-precision activations, we set the bitwidth of each layer so that all feature maps have the maximum feature map size, with a maximum bitwidth of 32. We show results in Figure 7. For Resnet50, the model with ∼400KB maximum feature map size corresponds to a uniformly quantized model with 4 bit activations.
Observation 7: Post-training quantization methods that use labeled data can perform equally well with pseudolabels.
While we use labeled training data for learning mixedprecision bit allocations, it is possible to generate labels using the floating point baseline, and train the model against those pseudolabels. In Figure 8, we show that the quan-   tized model accuracy is close to identical with ground truth labels compared to pseudolabels. The accuracy of such approaches may be further improved by using soft pseudolabels as in knowledge distillation or self training. This demonstrates that the data constraints of post-training quantization, e.g. whether using synthetic data or pseudolabels instead of ground truth training data, may be separated from the quality of quantization algorithms. We evaluate the accuracy obtained by the pytorchcv model (baseline accuracy 77.7%) and torchvision model (baseline accuracy 76%) with varying weight bitwidth in Figure 9. By virtue of the pretrained accuracy, the 5 bit quantized pytorchcv model outperforms all pytorch quantized models. However, at extremely low bitwidths, pretrained models with higher accuracy may not always perform better.

Discussion
Given that the aforementioned confounding tradeoffs impact quantized model accuracy by >1%, which may be as large as the improvements of state-of-the-art methods, we encourage all quantization papers to discuss and report their design tradeoffs through quantization cards. In Table 3, we present an example of a quantization card for the method used in Observation 3 which has full precision residual and skip connections, an often overlooked design tradeoff that the quantization card clearly highlights. In Appendix A, we provide an example of a quantization card for the posttraining mixed-precision method used in this work.
These tradeoffs can inform design decisions for developers deploying quantized models on hardware. Depending on primary bottlenecks (compute, memory bandwidth, or memory capacity), and acceleration opportunities (arithmetic with 2-4 bit integers), developers can use quantization cards to quickly determine whether the design tradeoffs made by the work are applicable for their use case.
While these design tradeoffs are likely to manifest similarly for different methods, the magnitude of their impact may vary depending on the quantization method and precision setting. For example, symmetric quantization of weights is likely to increase the required bitwidth by 0.23 bits on average for methods that round weights to their nearest quantized value post-training [3], but may have a smaller impact for methods that do not round to nearest [36,30,18] or perform quantization-aware training [14,34]. This emphasizes the importance of discussing the interactions between design tradeoffs and novel methods; if throughput on some hardware is substantially improved by using signed arithmetic, then methods that are less impacted by symmetric quantizers are preferable.
We leave the creation of a benchmark to future work, after the design scenarios for quantization are better established. Such a benchmark might include different resource constraints and quantization implementations, and allow developers to supply quantized model weights and quantizer parameters for the design setting of their choice. This will facilitate clearer evaluation between different approaches to quantization.

Conclusion
In this work, we demonstrate that novel quantization papers evaluate their results subject to design tradeoffs that are independent of the proposed method, though these tradeoffs can impact quantized network accuracy as much as the methods themselves. These include a variety of design decisions, some even seemingly innocuous, such as whether to use asymmetric/symmetric quantizers, per-channel vs. pertensor quantization, quantized or full precision skip connections, the set of allowed bitwidths for mixed-precision quan-

Appendices A. Learning Bitwidths for Post-Training Mixed-Precision Quantization
We describe the method for learning bitwdiths for mixed-precision post-training quantization formally, adapted from [34]. We let B We parameterize the bitwidths continuously, but round them to the nearest integer during the forward pass, allowing gradients to flow backwards using the Straight-through-Estimator (STE) for all rounding [5]. Then, our method requires solving the optimization problem We note several key differences from [34]. First, we parameterize directly with respect to the number of bits. Unlike [34], we observe no difference in empirical performance between parameterizing with respect to the bitwidth and the scale, and find that parameterizing with respect to the bitwidth is more natural for per-layer bitwidth allocation when using per-channel quantization parameters. Second, the optimization problem is post training, and is consequently only with respect to B, not the parameters θ. Third, because the number of samples is limited, we only train minimum and maximum clipping range for per-tensor quantization of activations, and not for per-channel weight quantizers. Training both may require hyperparameter tuning of learning rates for different parameter groups, as in [18]. We evaluate the model with the highest exponential moving average train accuracy that meets the constraints during training because the discrete nature of the optimization problem means that the model does not always meet the constraints at the final training step.
We show a quantization card for this method in Table 4.

B. Experimental Details
Unless otherwise stated, all experiments use uniform, asymmetric, static quantization, with per-tensor quantization of activations and per-channel quantization of weights. Baseline accuracy: 76% | Quantized accuracy: Table 2   Table 4. Quantization Card for Mixed-Precision Post-Training Quantization Skip and residual connections are quantized. Average pooling is performed with high precision accumulators and the outputs are requantized. We use the torchvision model with 76% baseline validation accuracy. When quantizing activations to low precision uniformly ( Figure 2, Figure 5), we use percentile clipping of activations at the 99.99th percentile, as in [23,27]. The first and last layer are kept at 8 bits, and all weights are 8 bits. 320 unlabeled examples are used for percentile clipping. Experiments in Figure 2 quantize skip connections but not residual connections, while experiments in Figure 5 vary the strategy for quantizing residual and skip connections.
For low precision weight quantization when using uniform precision (Figure 9, Figure 4), we use min/max calibration of activations and weights. The first and last layer are kept at 8 bits, and all activations are 8 bits. 1024 unlabeled examples are used for calibration.
Experiments with a maximum feature map constraint (Figure 7) keep the last layer at 8 bits, but quantize the first layer to low precision since the outputs of the first layer have the maximum feature map size. We use percentile clipping for activations.
For learned mixed precision of weights only (Figure 3, Figure 6), we use min/max calibration of activations and allow the bitwidths of all layers, including the first and last, to be any integer. Compression ratios include the first and last layer. All activations are 8 bits. For the bitwidth subset in Figure 6, 2, 4, and 8 bits are allowed. Mixed precision allocations are trained on 1024 labeled data examples for 1500 steps with a batch size of 32, which takes <20 minutes on a NVIDIA V100 GPU. Parameters are trained with the Adam optimizer [21] with a learning rate of 0.01.
When learned mixed precision of both activations and weights (Table 2), we use min/max calibration of weights, and allow the min/max for activations to be learned jointly with bit allocations. All integer bitwidths are allowed, and compression ratios include the first and last layer. For experiments when the compression ratio includes a 8-bit first and last layer, the bitwidth of the first and last layer is fixed. We use the same training setup as for mixed precision weights.