{'fraction_non_alphanumeric': 0.0365984209822026, 'fraction_numerical': 0.012712431419777867, 'mean_word_length': 4.818217205138186, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Transform-Based Feature Map Compression for CNN Inference

To achieve higher accuracy in machine learning tasks, very deep convolutional neural networks (CNNs) are designed recently. However, the large memory access of deep CNNs will lead to high power consumption. A variety of hardware-friendly compression methods have been proposed to reduce the data transfer bandwidth by exploiting the sparsity of feature maps. Most of them focus on designing a specialized encoding format to increase the compression ratio. Differently, we observe and exploit the sparsity distinction between activations in earlier and later layers to improve the compression ratio. We propose a novel hardware-friendly transform-based method named 1D-Discrete Cosine Transform on Channel dimension with Masks (DCT-CM), which intelligently combines DCT, masks, and a coding format to compress activations. The proposed algorithm achieves an average compression ratio of 2.9x (53% higher than the state-of-the-art transform-based feature map compression works) during inference on ResNet-50 with an 8-bit quantization scheme.


I. INTRODUCTION
Convolutional neural networks have been widely applied in a variety of domains, which achieve great performance in many tasks including image classification [1], object detection [2], and natural language processing [3]. With the evolution of the CNN models [4,5,6], more complicated structures with a larger number of parameters promise an improvement of predicting accuracy. However, the high power consumption and the unacceptable latency caused by deeper CNN models hinder their implementation of the mobile devices and Internet-of-Thing (IoT) applications on Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC) with low hardware budget.
To solve this problem, researchers usually try to reduce the computational cost of models by pruning convolutional filters [7], such as Neural Architecture Search (NAS) [8,9]. However, data transfer is another factor that brings high power consumption and latency in the CNN accelerator. When deep CNN models are deployed on FPGA, all feature map data and filters cannot be loaded on-chip at the same time. As a result, the large data of feature maps on CNN have to be repeatedly transferred between on-chip and off-chip memory, which greatly increases the power consumption and the data transfer bandwidth. Therefore, it is a topic worth exploring to reduce this power consumption and latency by compressing feature maps to improve the performance of CNN models on specialized hardware [10].
Many CNN feature map compression algorithms that use the sparsity of activation have been proposed over the last few years. [11][12][13][14][15] focus on designing hardware-friendly compression encoders and decompression decoders or reducing the calculation cost of sparse data by using a special storage format. [16][17][18][19] design algorithms to compress data transferred between GPU and CPU memory during training.
In this paper, we design transform-based methods that improve the sparsity of activations before data are sent into sparsity-based compression modules and further increase the compression ratio for CNN inference.
For fused-layer and quantized CNN models [20] deployed on hardware accelerator, we propose different methods for activations to deeply exploit and improve feature map sparsity, including:

•
We observe the specific 2-D frequency domain information of activations, which demonstrates the fact that we can use 2D-DCT to compress feature maps in earlier layers of CNN. Then we combine Approximate Sparsity Preprocessing (ASP) and lowbit quantization to compress feature maps in the later layers, which achieves a compression ratio of 2.6× (+30% over [16]).
• By exploiting the difference in the sparsity of activations in different layers, we propose DCT-CM to compress each activation transferred, which achieves a 53% higher compression ratio than [16].
However, the sparsity of feature maps is dynamic, which limits the compression ratio of prior methods. A few more methods change CNN`s structure to compress feature maps but they introduce much accuracy loss [18,29]. Besides, when compression algorithms are implemented, non-dictionary is regarded as a symbol of hardware-friendly by the majority of researches [10].
Among the non-dictionary compression works, Zero-Value Compression (ZVC) is a simple and highly effective approach based on frequent-value compression that uses Bitmap to store the compressed data. It can achieve a compression ratio of 2.5× for 60% sparse data. Thus, ZVC is widely-used in different compression-based accelerators: [18,30].

B. JPEG for ACTivations (JPEG-ACT)
JPEG [31] is an image compression based on the transformation of the frequency domain by using DCT to save important low-frequency components. JPEG-ACT [16] tries to use the same transformation and similar coding format to compress feature maps during training. It introduces trainable quantization matrixes to compress the wide variety of frequency domain information of feature maps on CNN. JPEG-ACT achieves a higher training accuracy and a higher compression ratio among the training accelerators by compressing activations [17][18][19]. However, in an inference accelerator, this method will bring extra transfer and computational cost.
For this work, we introduce the hardware-friendly DCT and ZVC, then we fuse them with the mask method which stores a few additional data to achieve a higher compression ratio and a lower hardware cost at the same time.

III. COMPRESSION ALGORITHM
To compress feature maps for inference, we propose the following transform-based methods to exploit data sparsity in fused-layer CNN [20]. Then we use a sparsity-based compressor, ZVC, to complete the compression procedure. Because the compression algorithm will be deployed on a hardware accelerator, the hardware-friendly and widely-used DCT algorithm becomes our choice.

A. 2D-Discrete Cosine Transform on Earlier Activations
2D-DCT is a common algorithm to process figures in frequency-domain. Inspired by JPEG compressor, we compress feature maps by dicing them into 8×8 patches, applying 2D-DCT (1), where X for original data and Y for transformed data, and then quantizing and compressing. After transfer, data need decompression and IDCT (Inverse-DCT) (2). However, prior work indicates that feature maps have different frequency information from visual images [32]. Therefore we have to deal with the frequency domain information of special dimension carefully, instead of cutting high-frequency components blindly.
Because the frequency domain entropy of earlier activations which always require more storage is lower [16], 2D-DCT with quantization is effective to compress them. Then we observe that the later activations are so sparse ( Fig.  1) that lower-bit quantization brings a high enough compression ratio. By combining these two ways as Fig. 2, the compression ratio of feature maps can be improved with the validation accuracy kept for image classification tasks.

Y = A T XA
(1) When deploying this method on accelerators, the encoder and decoder modules should transfer activations to registers in shape 1×8×8 to achieve higher throughput. Besides, it is not recommended to read in data from the same channel repeatedly. E.g., it is proper that convolutional filters slide in the channel dimension of input.

B. 1D-Discrete Cosine Transform on Channel Dimension with Masks
The 2D-DCT in part A exploits the 2-D frequency domain distinction between different layers to compress feature maps, but the reusability of the transform module is poor. To solve this problem, we propose DCT-CM, which uses the potential redundancy in the channel dimension of feature maps. DCT-CM transforms the feature map on the channel dimension and compresses it by dropping the high-frequency component.
The first step of the DCT-CM is using 1D-DCT on the channel dimension of each 8×1×1 patch of feature maps as (4) , where A is the same as (3). Then, DCT-CM uses masks changing with the convolutional block on the transformed patches, and the shape of the mask is the same as patches, 8×1×1. Masks with different valid regions will screen the transformed feature map and only save the low-frequency component as Fig. 3.

Y = AX
(4) Although there is redundancy [33] and vacancy in feature maps, 1D-DCT cannot directly exploit them to compress feature maps. We observe that the distribution of values in the channel dimension is random, which implies that the frequency domain entropy is high. We have to reorder the data after transformation by an extra mask, so that the local channels of the model will be low-frequency during training, then we can improve the sparsity of activation. As a result, the DCT-CM will restructure the filters on CNN, then this algorithm has to be used at the start of training instead of finetuning.
The 2D-DCT has two disadvantages in the inference that it cannot be deployed on small-scale feature maps and it only shows a high compression ratio on earlier blocks in image processing tasks. In contrast, DCT-CM can be used on any activation, which brings much more hardware reusability. Furthermore, the shape (or the length) of masks can be adjusted, e.g. from 8 to 16 or 4. The selection depends on the balance of computation cost and compression ratio. Besides, the valid region of mask groups can be also set by NAS.
By using the high sparsity of feature maps, [33] chooses feature selection on high dimensional vectors and [18,29] change the structure of CNNs. On the contrary, the DCT-CM transforms the feature map on the channel dimension instead of the special dimension. Therefore, this algorithm reorders the channels of activations during training and provides static sparsity for the compression module.
Considering the computational cost, compared with 2D-DCT, decompression modules of DCT-CM will read fewer values, since parts of the data are set to zero by the mask. Thus, the DCT-CM which saves low-frequency information regularly can reduce both the bandwidth and the inversetransform computational cost. DCT-CM operates every single pixel on the channel dimension of feature maps and uses fewer matrix operations for each unit (4), (5).
Furthermore, the transformation matrix A in (5) can be fused into next convolutional layers by the linear transformation on weights. Then, we use the fact that ZVC describes both the compression algorithm and the data flow of compressed feature map in convolutional filters. As a result, we can cut off the extra computational budget caused by IDCT. E.g., we take a transformed slice, Y freq-domain ∈ℝ 8×1×1 , and use (6) to get Y reshape . A block of weights in convolutional kernels is W ∈ ℝ × , Y out is calculate by (7). Then we only need to save W * of (8) for the inference and reduce inversetransform computational budget.
Therefore, DCT-CM can provide more flexibility in convolutional architecture design and higher throughput than 2D-DCT. Furthermore, the DCT-CM algorithm introduces a little computation of matrix multiplication compared with the convolution. E.g., for each block and its outputs in c4 in ResNet-50 [4], 0.2G MACs (Multiply Accumulates) is needed by convolutional layers to compute the output feature maps, but DCT-CM only need 0.4M MACs (0.2%) with the mask type 2/8 after applying the further optimization that skips the zero computation caused by the mask.

C. Approximate Sparsity Preprocess and L1-Loss
Feature maps can be sparse after extracted from ReLU in neural networks. Furthermore, small values in these feature maps can be dropped to increase the sparsity immediately. It is noticeable that this method is different from lower-bit quantization because the precision of large values is still kept the same as the original quantization.
The proposed preprocessing algorithm, ASP, (9) is very simple, which does not need to constrain the data reading. The parameters of the new model using this method can be set by fine-tuning.
i _ _ f el _ se Act value = 0 Act value < Threshold Act value (9) Although ASP is a lossy algorithm, it keeps good accuracy. This straightforward sparsity algorithm runs without a homologous decoder. Thus, we insert ASP into 2D-DCT and DCT-CM to further improve the sparsity of feature maps.
Another widely-used way to increase the dynamic sparsity of feature maps is to regularize activations by extra loss functions, such as L0, L1, or smooth L1. The effect of L1 loss in our work will be discussed in Section IV.

A. Experimental Setup
We train an 8-bit quantization ResNet-50 [4] as a baseline on CIFAR-10 [34]. Because the proposed algorithms are sparsity-based, we calculate the compression ratio by ZVC encoder. All the feature maps are extracted after the last ReLU activations of each block in ResNets.

B. Selecting Parameters and Combinations
During the experiments, we aim at achieving higher compression ratio as much as possible within 0.5% accuracy drop in image classification. Table Ⅰ. shows that the proposed DCT-CM can express a higher compression ratio of 2.9× with less accuracy dropped of 0.39%. Table Ⅱ. lists representative settings and the results of our experiments. When deploying 2D-DCT algorithms, we apply 2D-DCT to the first layer (stage c1 in ResNet-50, represented by parameter '1') or earlier blocks (stage c1 and c2 in ResNet-50, represented by '1 & 2 blocks') but apply low-bit quantization to the rest. Besides, 2D-DCT performs well on accuracy rather than compression ratio, which implies it can be used on complex tasks. The L1 loss cannot improve the sparsity further based on proposed methods.
Though ASP shows high sparsity and compression ratio, it works worse with a higher threshold. Therefore, we infer that ASP is not qualified for the model quantized in lower-bit but can be used as an optional plug-in with transform-based algorithms. Differently, the proposed novel method, DCT-CM, can express both high validation accuracy and high compression ratio in a variety of parameters with appropriate masks (M-1 and M-2 in Table Ⅱ. for different mask settings, which are the same as them showed in Fig. 1 Our experiments indicate the flexibility of the parameter selection of DCT-CM, which implies its prospect in complicated tasks. Besides, by combining higher-bit and ASP, DCT-CM can improve the compression ratio and also maintain accuracy.

V. CONCLUSION & FUTURE WORK
In this paper, we propose novel transform-based and sparsity-based algorithms DCT-CM to reduce the data transfer bandwidth and power consumption for CNN inference. By combining it with ZVC, we achieve a compression ratio of 2.9× with negligible accuracy dropped on a CNN model with an 8-bit fixed-point quantization scheme for inference. Designing a hardware accelerator architecture for DCT-CM and combining NAS into masks selection are our future directions.