{'fraction_non_alphanumeric': 0.03599732645851236, 'fraction_numerical': 0.008498042585696553, 'mean_word_length': 4.882336422353272, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Semantic-assisted image compression

Conventional image compression methods typically aim at pixel-level consistency while ignoring the performance of downstream AI tasks.To solve this problem, this paper proposes a Semantic-Assisted Image Compression method (SAIC), which can maintain semantic-level consistency to enable high performance of downstream AI tasks.To this end, we train the compression network using semantic-level loss function. In particular, semantic-level loss is measured using gradient-based semantic weights mechanism (GSW). GSW directly consider downstream AI tasks' perceptual results. Then, this paper proposes a semantic-level distortion evaluation metric to quantify the amount of semantic information retained during the compression process. Experimental results show that the proposed SAIC method can retain more semantic-level information and achieve better performance of downstream AI tasks compared to the traditional deep learning-based method and the advanced perceptual method at the same compression ratio.


INTRODUCTION
With the explosion of visual data on the internet, image compression has becoming a significant and fundamental task especially for caching and communication. There is a plethora of prior art on image compression including conventional methods such as JPEG and JPEG2000, and methods based on deep learning. Conventional methods compress images separately in transform, quantizer, and entropy code [1], which is inefficient. In contrast, deep learning-based compression methods are powerful due to the joint optimization of the entire compression model and excellent learning ability. Deep learning-based image compression have been explored by convolutional autoencoder (CAE) [2], recurrent network (RNN) [3], and generative adversarial networks (GAN) [4].
However, existing image compression methods aimed at maintaining pixel-level consistency. With the development  Images contain pixel-level information, and go through the downstream AI task, feature-level and semanticlevel information can be obtained. Semantic-level information is the meaning finally understood by the downstream AI task, which is contained in perceptual results, such as locations and confidence scores.
of computer vision, a large number of compressed images need to be understood by downstream AI tasks such as image recognition, object detection, etc. Image compression should have both high visual quality and high performance of downstream AI tasks. Solely maintaining pixel-level consistency cannot guarantee the task performance.
To this end, Yang et al. [5] proposed a discernible image compression method aiming at maintaining feature-level consistency of downstream AI tasks. Although features are sensitive, using feature maps as loss functions is still reliable, which is demonstrated through solid experiments in [5]. To further illustrate the reliability, Table 1 shows that the featurelevel distortion (mean square error of feature maps) decreases with the decrease of compression degree.
However, the performances of downstream AI tasks are ultimately determined by semantic-level information rather than feature-level information. This paper utilizes the fact that perceptual results contain all semantic-level information understood by the downstream AI task. Fig.1 illustrates the relationship among pixel-level, feature-level and semantic-level information. Although semantic-level information is further extracted from feature-level information and good feature- level information is more likely to extract good semanticlevel information [6], they are essentially different. Featurelevel information is the intermediate output of the downstream AI task, while semantic-level information is the final meaning understood by the downstream AI task. Thus, semantic-level information can influence AI task's performance directly. Taking object detection task as an example, the task performance mean average precision (mAP) is determined by the semantic-level information, which is contained in perceptual results (bounding boxes and confidence scores). Therefore, different from maintaining feature-level consistency, maintaining semantic-level consistency during the compression process has the potential to further improve the performance of downstream AI tasks.
In this work, we propose a Semantic-Assisted Image Compression method (SAIC) that maintains semantic-level consistency during compression. The purpose is retaining semantic-level information of downstream AI tasks during compression and consequently obtain good task performance. The innovation is that utilizes semantic-level information of downstream AI tasks and closely combining the compression task with the downstream AI task. The main contributions of this work are: • We propose a SAIC method, in which the semanticlevel information of downstream tasks is utilized to assist image compression. • We propose a gradient-based semantic weights mechanism (GSW) to obtain semantic-level importance, which directly considers downstream AI tasks' perceptual results. • We propose a semantic mutual information metric (SI) to quantify the semantic-level distortion during compression process for specific downstream AI task.

RELATED WORK
Image Compression. There are several image compression methods based on deep learning, such as RNN-based networks [3], CNN-based networks [7] and generative adversarial networks (GAN) [4], which use pixel-level difference as distortion and do not consider downstream tasks. In addition, some advanced works took content information [8] and task information [9] [5] into consideration. In particular, Li et al. [8] considered edges and textures information without considering the task performance. Patwa et.al. [9] simultaneously accomplished classification and decoding using the same compact feature representation, which may hurt the task performance. Yang et al. [5] utilized feature-level information in loss function. In conclusion, these works do not consider semantic-level information of the downstream AI task. The proposed SAIC focuses on retaining semantic-level information of downstream AI tasks during compression, and strive for satisfying task performance. Interpretable CNN. We introduce interpretable CNN to extract semantic-level information. Selvaraju et al. [10] proposed a Gradient-weighted Class Activation Mapping (Grad-CAM) method, which can make visual explanation for perceptual results. The factors that determine perceptual results inspire us to extract the useful semantic-level information.

METHOD
This section elaborates the proposed SAIC, which can retain semantic-level information of downstream AI tasks during compression process by using semantic-level loss function. As illustrated in Fig.2, the proposed SAIC architecture is composed of two modules, i.e. semantic-level loss function calculation module and encoder-decoder based compression network module. In particular, we first utilize semantic-level loss function calculation module to calculate the semanticlevel loss function L SAIC . Then we use the semantic-level loss function L SAIC to train the encoder-decoder based compression network module. As shown in Fig.2, the original image x and the compressed image x first pass a pre-trained CNN to obtain the feature maps F (x) and F (x ). Then, we calculate the semantic-level weights W by GSW mechanism using the gradient from perceptual results to feature maps. Next, we use F (x), F (x ) and W to compute the semantic-level loss function L SAIC . Finally, we utilize the semantic-level loss function L SAIC to train the encoder-decoder based compression network.

GSW Mechanism
GSW mechanism can obtain semantic weights, which is used to construct SAIC method. Semantic-level information is contained in perceptual results, and task performance is determined by perceptual results. GSW mechanism uses the gradients of perceptual results respect to feature maps, which can naturally represent the contribution of feature maps respect to perceptual results.
We first pre-train a feature extraction network of downstream AI task to provide input for GSW mechanism, which can be expressed as: where f k (k ∈ {1, 2, ..., K}) is the k-th feature map, θ 1 is the fixed parameter of feature extraction network, and M , N and K represent the width, height, and total number of the feature maps, respectively. Then, semantic-level information is further extracted from feature-level information by post-processing network. In this process, perceptual results

Encoder-Decoder Based Compression Network
Semantic Loss LSAIC  For example, the perceptual results y are confidence scores in classification task, while confidence scores and localization information in object detection task. Obviously, as the input of semantic extraction network, feature maps F have different semantic-level importance degrees to obtain the semanticlevel perceptual results for downstream AI task. Then, the gradients of perceptual results respect to feature maps are used to quantify the semantic-level importance degrees, which is a matter of course due to the meaning of gradient. We compute the gradient of c-th perceptual result y c with respect to the k-th feature map f k , i.e. ∂y c ∂f k . Then use global-average-pooled over the width M and height N dimensions (indexed by m and n respectively) to obtain the importance weights w c k by: where f k ∈ R M ×N . To obtain the semantic weights of the whole perceptual results y, we compute the average value: Thus, we obtain channel-wise semantic weights W = {w 1 , w 2 , ..., w K } ∈ R K , where w k (k ∈ {1, 2, ..., K}) represents the semantic importance degree of k-th channel's feature map. The channel-wise semantic weights can be obtained under any dimension of perceptual results.
However, the value of W is too small to be directly used in the loss function, since it may cause the slow convergence. Therefore, we utilize parameter τ , and map the weights W to the weights W by: where τ is a temperature hyper-parameter, and r is a constant. W is appropriate for semantic-level loss function L SAIC . The temperature hyper-parameter τ can control the tightness of the semantic weights' distribution. Finally, we multiply a constant r to make the final semantic-level loss value at a reasonable magnitude, which will not affect gradient updates. Note that due to GSW mechanism, semantic weights are taskspecific, so SAIC is task-specific.

Semantic-level Loss Function calculation
We introduce this section to formulate semantic-level loss function L SAIC for the image compression network. Generally, the loss function of traditional deep learning-based image compression network of encoder-decoder structure can be written as: where B (batch size) is the number of images per iteration, b is the index of images, is the encoder network with parameter θ 2 for compressing the given image is the decoder network with parameter θ 3 for recovering the compressed latent features to images x b . The compressed image can be written as: For the sake of convenience, we use e = E(x), q = Q(E(x)), d = D(B(E(x))) to represent the output of encoder, quantizer and decoder, respectively. Note that we use binary quantization to map the encoder output e to 0 or 1, which can be expressed as: Q(e) = 1, e > 0.5, 0, e ≤ 0.5.
1: Pre-train the downstream network with parameter θ 1 , and fixed θ 1 in the following operation. 2: Obtain the semantic weights W using GSW. 3 Update θ 2 and θ 3 according to L SAIC (θ 2 , θ 3 ) 14: end while 15: return The optimal compression model The pre-trained network extract feature-level information from original images and compressed images respectively by: where k-th feature map of original image b is f b k and k-th feature map of compressed image b is f b k , k ∈ {1, 2, ..., K}. Then, we utilize the channel-wise K-dimension semantic weights W = {w 1 , w 2 , ..., w K } ∈ R K obtained by GSW to weight image's feature maps as image's semantic-level information. Therefore, the loss function of SAIC can be written as: We summarize the steps of SAIC as Algorithm 1. In step 1, we pre-train the downstream AI network. In step 2, we obtain the semantic weights. In step 3-15, we compute L SAIC (θ 2 , θ 3 ) and use it to train the compression network.

SI Estimation
To quantify the semantic-level distortion during compression process for specific downstream AI task, we propose a semantic mutual information metric (SI). SI is the mutual information of all perceptual results, which contain all semantic-level information of the downstream AI task. Let the perceptual results of the original image x b be y b , and the perceptual results of compressed image x b be y b . However, it is challenging to estimate SI between y b and y b , since the entropy of the original image dataset is mathematical intractable. We utilize CLUB [11] to estimate SI due to its excellent accuracy.
In particular, we first input y b and y b to train a SI estimation network, from which we can obtain the mean and variance of y b . Then we can compute the conditional probability p(y b | y b ) using the mean and variance. Finally, we compute SI using p(y b | y b ) by:

EXPERIMENT 4.1. Dataset
For classification task, we carry out experiments on STL and ImageNet datasets. STL dataset [12] has 10 classes, and each class contains 1300 images in PNG format. In terms of ImageNet [13], we use about 9000 images in JPEG format. For object detection task, the experiments adopt Pascal VOC dataset [14], which contains 27088 images in JPG format.

Evaluation Metrics
We use semantic mutual information (SI), accuracy (ACC), F1-score and mean average precision (mAP) to evaluate the semantic-level distortion. We use peak signal to noise ratio (PSNR), structural similarity index measure (SSIM) and mean square error per pixel (MSE) to evaluate the pixel-level distortion. We use compression ratio to evaluate the compression degree.

Comparison Methods
We use two baseline schemes for comparison: traditional deep learning-based image compression method (TDIC) and advanced perceptual image compression method (APIC). Note that the proposed SAIC and the two baseline schemes have different loss functions. For fair comparison, we use the same experiment conditions except for the loss function, including pre-trained downstream AI task network, the encoder-decoder based compression network structure and so on. TDIC, APIC and SAIC use pixel-level loss function, feature-level loss function [5] and semantic-level loss function, respectively.

Implementation Details
To reduce training costs and promote extensibility, we use the two-stage approach to train the APIC and SAIC model. In the first stage, the TDIC model is pre-trained for 3-4×10 5 steps on different datasets with a batch size of 32 and a learning rate of 1×10 −5 . In the second stage, we finetune using APIC and SAIC methods for 2-3×10 4 steps with a batch size of 32 and a learning rate of 1×10 −5 . For fair comparisons, to let the data load sequence be random and consistent, we fix random seed. Thus, the experiment is stable and repeatable. See appendix for specific experimental conditions.    Task   Table 2, Table 3 and Table 4 show classification results. It is apparent that the proposed SAIC can always obtain better SI, ACC and F1-score in different datasets, different compression ratios and different classification network structures. As we can observe, SI, ACC and F1-score have the same trend, and ACC is very close to F1-score. SAIC can retain more useful semantic-level information in the compressed image due to the design of semantic-level loss function, and deservedly can obtain competitive SI, ACC and F1-score value. In addition, the conventional compression metrics PSNR, MSE and SSIM are as good as APIC method. These results suggest that without damaging the traditional indicators, we significantly reduce the semantic-level distortion and promote the ACC and F1-score of the downstream AI task. As shown in Fig.3, the four sub-figures have little visual difference, while have totally different perceptual results for downstream AI task. See appendix for more examples. Some minor distortions caused by compression can lead to error perceptual results in downstream AI task. TDIC and APIC make a mistake since pixel-level and feature-level consistency cannot guaranty perceptual results. In contrast, the proposed SAIC directly focus on semantic-level consistency, thus can make right perceptual results and has the potential to own a better task performance.

Parameter Analysis
Ablation Experiments. To evaluate the influence of GSW mechanism, we conduct ablation experiments. GSW get semantic weights of feature maps, which are used in L SAIC  Fig. 4. Classification results on ImageNet dataset with 0.5 bpp using different hyper-parameter τ .
to retain more semantic-level information. We conduct ablation experiment when feature maps have the same importance degree and no semantic difference. In that case, semantic weights equal to 1 for all feature maps. Thus, SAIC degenerates into APIC. APIC is the special case of SAIC when semantic-level of feature maps are equally important. That is to say, APIC is SAIC without GSW. Impact of Hyper-parameter. To evaluate the influence of temperature hyper-parameter τ , Fig.4 shows the performance of SAIC with different temperature hyper-parameters τ . It is significant in Fig.4 that with τ increases, the overall variation trend of SI and ACC is first up, then down, then up and down again. This is because the hyper-parameter τ can control the degree of dispersion of semantic weights distribution, and extreme concentrated (τ = 1, τ = 10000) or decentralized (τ = 2000) semantic weights can degrade the performance. Excessively concentrated semantic weights almost the same importance degrees in different channels, and SAIC is close to APIC. Excessively decentralized semantic weights are equivalent to deleting some feature maps. Only appropriate degree of dispersion can have better performance improvement. The optimal value of τ varies with the model structure and the dataset. See appendix for semantic weights

Experiments on Object Detection Task
The SAIC method has generalization ability, and the images compressed by SAIC can be used for a variety of downstream AI tasks. To verify generalization performance, the images compressed by SAIC can be applied to the object detection tasks. We select RFBNet [15] trained on VOC0712 to conduct the object detection task. Table 5 shows the mAP values of original and compressed images (bpp=0.5) using different methods on VOC 2007 validation. As shown in Table 5, the task performance (mAP) of APIC and SAIC is comparable and higher than TDIC. To further improve task performance on object detection, we can utilize semantic-level information of object detection task to train the compression network in the future.

Compression Ratio
Traditional image compression uses rate-distortion to evaluate compression performance. Compression ratio represents compression degree and pixel-level distortion (e.g.,PSNR, MSE) represents distortion degree. We use semantic-level rate-distortion to evaluate compression performance. Compression ratio represents compression degree while SI and ACC represent semantic-level distortion. Table 6 shows SI and ACC of TDIC, APIC and SAIC methods with 0.125 bpp, 0.25 bpp and 0.5 bpp. Thanks to the quantizer, accurate compression ratios can be calculated. Different compression ratios can be obtained by controlling the size of quantized feature maps q. As shown in Table 6, at the same compression ratio, SAIC can get less semantic-level distortion. In turn, at the same semantic-level distortion, SAIC can compress more. As the compression ratio shrinks, the performance improvement getting higher and higher. That's because with the degree of compression decreasing, the compressed image is less distorted and leaves less room for improvement. Some images can be correctly perceived without the assistance of semanticlevel information.

CONCLUSION
This work proposed a novel SAIC method, which takes downstream AI tasks into consideration. In particular, the proposed SAIC innovatively aims at maximizing the semantic-level information required by downstream AI tasks during compression process, so as to improve the downstream AI tasks' performance. A new metric SI has also been proposed to quantify the semantic-level distortion during image compression. Experimental results show that SAIC can achieve 10.16% and 1.66% higher ACC values on STL dataset with 0.125 bpp than TDIC and APIC, respectively. The proposed SAIC method can take into account human visual experience and machine perception performance, and can be used for a variety of intelligent tasks. In the future, it can be deployed for other applications such as denoising and super-resolution.