{'fraction_non_alphanumeric': 0.03754106053496011, 'fraction_numerical': 0.014117002972000626, 'mean_word_length': 4.977793361383824, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Disparity-based Stereo Image Compression with Aligned Cross-View Priors

With the wide application of stereo images in various fields, the research on stereo image compression (SIC) attracts extensive attention from academia and industry. The core of SIC is to fully explore the mutual information between the left and right images and reduce redundancy between views as much as possible. In this paper, we propose DispSIC, an end-to-end trainable deep neural network, in which we jointly train a stereo matching model to assist in the image compression task. Based on the stereo matching results (i.e. disparity), the right image can be easily warped to the left view, and only the residuals between the left and right views are encoded for the left image. A three-branch auto-encoder architecture is adopted in DispSIC, which encodes the right image, the disparity map and the residuals respectively. During training, the whole network can learn how to adaptively allocate bitrates to these three parts, achieving better rate-distortion performance at the cost of a lower disparity map bitrates. Moreover, we propose a conditional entropy model with aligned cross-view priors for SIC, which takes the warped latents of the right image as priors to improve the accuracy of the probability estimation for the left image. Experimental results demonstrate that our proposed method achieves superior performance compared to other existing SIC methods on the KITTI and InStereo2K datasets both quantitatively and qualitatively.


INTRODUCTION
Recently, stereo images have been widely used in various fields, such as 3D movies, virtual reality, autonomous driving and so on. These large numbers of high-quality stereo images pose new challenges for data transmission and storage, which require efficient image compression methods to reduce the cost.
Image compression is a fundamental and vital research topic in the field of multimedia. After decades of research, a lot of traditional image compression standards have been proposed, such as JPEG [57], JPEG2000 [14], BPG [10] and VVC-intra [53]. Typically, they follow the pipeline of three hand-crafted modules: transformation, quantization and entropy coding. Recently, benefiting from the end-to-end optimization, deep neural network (DNN) based image compression methods have achieved promising results [3,4,12,21,23,32,45,47,58,65].
A stereo image pair consists of left and right images, which are captured by a stereo camera from both views at the same time. For stereo images, a simple idea is to compress each image separately using a single image compression method. However, due to the large overlapping fields of view between the left and right cameras, quite a lot of similar information inherently exists between stereo images. This idea ignores the high correlation between views and wastes a lot of bitrates, which is rather unwise. Therefore, it is necessary to design a compression method that can make full use of the inner relationship and minimize the redundancy between views as much as possible.
Stereo image compression (SIC) compresses the left and right images jointly, aiming to achieve high compression ratios for both two images. At present, there have been many studies focusing on SIC. In many traditional SIC methods [36,44], the disparity map is often compressed additionally as important side information to reduce  Figure 1: Brief framework of our proposed DispSIC. A threebranches is adopted, which encodes the input right image, the disparity map and the residuals of the left image respectively.
the bitrates. Given a reference view, they use the disparity compensated prediction to encode other views. However, traditional methods minimize the rate-distortion loss based on hand-crafted features, which limits their compression efficiency. Recently, deep learning-based SIC methods have achieved better performance than traditional ones. Liu et al. [35] proposed the first deep stereo image compression network, namely DSIC, which densely warps the features from the left image to the right at all levels in the encoder and decoder to exploit the content redundancy between the stereo pair. When training and testing, DSIC requires to construct cost volume and aggregate the cost multiple times, which has high computation complexity. After that, Deng et al. [16] propose an efficient stereo image compression based on homography transformation (HESIC), which reduces the computation complexity and achieves better compression performance. In HESIC, the left image is spatially transformed by a 3 × 3 homography matrix and the residuals between the right and the transformed left image are encoded. However, the homography matrix is used to describe the relationship between two images of the same planar surface in space. Most stereo image pairs have difficulty meeting this condition, which will make the residuals in HESIC very large. Compared with the homography matrix, the disparity map establishes a more accurate pixel-to-pixel correspondence between stereo images, which can significantly reduce the residual information. Moreover, as a simple 1D data flow, the disparity map is easily compressed. In this paper, we propose a disparity-based deep neural network for stereo image compression, namely DispSIC, which combines the disparity compensated prediction in traditional methods with a deep learning framework. As Fig. 1 shows, we replace the homography matrix in HESIC [16] with the disparity map, which is output by a stereo matching model. Unlike the traditional SIC methods using disparity compensated prediction, we train the stereo matching model and the compression model jointly with the rate-distortion loss. By end-to-end optimization, the stereo matching model learns how to provide more accurate disparity compensated prediction, and the compression model learns how to trade off the transmission cost and gain of the disparity map. The two models work toward the same objective i.e. reconstructing higher quality images with fewer bitrates.
In conclusion, the main contributions of this paper are as follows: • We propose a novel disparity-based deep neural network for stereo image compression, namely DispSIC, which uses the disparity map to explicitly represent the relationship between the left and right images. • We design a conditional entropy model with aligned cross-view priors, which can model the probability of the left image more accurately. • The experimental results show that our proposed method significantly outperforms the state-of-art deep stereo image compression methods, and saves around 25% bitrates compared to the latest method HESIC [16] with similar image quality.

RELATED WORK 2.1 Single Image Compression
Existing traditional image compression standards include JPEG [57], JPEG2000 [14], BPG [10] and VVC-intra [53]. The pipeline of these methods typically consists of transformation, quantization, and entropy coding. Although these traditional methods can achieve high compression efficiency, they heavily rely on prior knowledge to design hand-crafted modules. In recent years, the DNN-based image compression methods [3,4,12,21,23,32,45,47,58,65] have achieved great success with impressive performance. Unlike traditional methods, they jointly optimize all modules in an endto-end manner. For the network architecture, some early works used the recurrent neural networks (RNNs) to achieve progressive compression [52,56]. Then, most works used the convolutional neural networks (CNNs) to build an auto-encoder style network. Most recently, there have been some works employing the invertible neural networks (INNs) [58] and transformer [47,65] architecture. Besides, some works [1,3,55] were proposed to solve the problem of non-differential quantization and rate estimation. Other works such as generalized divisive normalization (GDN) [3], attention mechanism [12,34], residual blocks [12], adaptive feature extraction models [38], adversarial training [2,43,48], and importance map [42,63] were focused on improving the image compression performance. Meanwhile, variable-rate model [13,15], scalable compression [25,41] were introduced to meet the practical application requirements.

Stereo Matching
Stereo matching is the process of finding pixels corresponding to the same 3D point in the scene, consisting of four steps: matching cost computation, cost aggregation, disparity optimization and postprocessing [50]. As we know, the core of SIC is to explore the inner correlation between the left and right images, in which stereo matching techniques play an important role. Traditional stereo matching methods can be roughly divided into three categories: local methods [8,49], global methods [30,31] and semi-global methods [22]. Local methods compute the matching cost on a block-by-block or pixel-by-pixel basis, which run fast but have low-quality results. Global methods first obtain an initial Leaky ReLU Leaky ReLU ෝ → ഥ → Figure 2: The overall network architecture of our proposed method (DispSIC). We compress the left and right images jointly, and use the disparity map to explicitly represent the pixel-wise correlation between views to save bitrates. C represents the concatenation operation, Q represents quantization.
disparity map through simple matching, and then iteratively update the disparity map by optimizing a pre-defined global energy function. These methods can get high-quality results but are timeconsuming. Semi-global matching (SGM) methods perform line optimization along multiple directions, which successfully combine local and global stereo methods and achieve a favorable trade-off between computing time and quality of the results. Although traditional methods have made great progress, they do not work well in regions of low texture, repeating patterns and thin structures. Nowadays, with the development of deep learning, CNN-based stereo matching methods [11,27,40] achieve promising performance even in the above challenging regions. In terms of the network structure, existing end-to-end stereo matching methods include 2D [33,51,60] and 3D [20,61] convolution-based neural networks, differing in the way of cost aggregation. In comparison, most of the 3D-CNN-based methods perform better but are more time-consuming. In terms of training methods, most studies [37,62] use the ground-truth disparity as supervision to train the network. However, given that the ground truth is not always available, some studies [46,64] warp the left image to the right view based on the estimated disparity map and compute the reconstruction error as supervision.

Stereo Image Compression
Traditional SIC methods [7,17,18,26,29,39,44] usually compress stereo images in the way of video compression, which regards the inter-view similarity between the stereo camera views as the temporal similarity between successive frames in the video. Given a reference view, many works [36,44] use the disparity compensated prediction to encode each view, which is similar to the motion-compensated prediction in the single-view video. These methods rely on hand-crafted features and design modules individually, which greatly limits their compression performance.
Recently, several DNN-based methods for SIC have been proposed, which outperform the traditional methods. Liu [35] proposed the first DNN-based stereo image compression framework (DSIC), which uses parametric skip functions to feed the disparity-warped features at all levels from the left encoder/decoder to the right one. Due to the dense warp scheme, it has high computation complexity. After that, Deng et al . [16] use a homography transformation to replace the dense warp, which reduces the computation complexity and achieves state-of-art performance. In HESIC [16], a 3 × 3 homography matrix is used to spatially transform the left image to the right view. And only the residuals between the original right image and the transformed left image are encoded to reconstruct the right image. However, only when two images are of the same planar surface in space, the homography matrix can provide an accurate match. So in most scenarios, the homography transformation will lead to large residuals in HESIC [16]. Fig. 2 shows the overall framework of our proposed DispSIC network. In general, the whole network adopts a three-branch structure, which is used to encode the right image, the disparity map and the residuals of the left image respectively.

Framework
Firstly, taking the original left and right images (denoted as and ) as input, we use a stereo matching model to estimate the disparity map (denoted as ). Then, and are compressed via two independent auto-encoders [45]. Based on the decoded disparity map^, the decoded right image^can be easily warped to the left view, obtaining the predicted left image^→ . After that, we concatenate the original left image and the predicted^→ as the input to the third auto-encoder [45], which learns to compress the residuals. Taking into account the correlation between and , we take the warped quantized latent of as priors for the entropy model of the latent of . At the decoder side, the decoded residuals are concatenated with the predicted^→ to reconstruct the final left image^.

Disparity-based Prediction
To fully explore the inner relationship between stereo images, we need to match the images or features first. DSIC [35] constructs cost volume at each feature level to match the features between stereo images, which works but has high computation complexity. HESIC [16] uses a 3 × 3 homography matrix to roughly match the stereo images, which is simple but inaccurate. As Fig. 3 shows, the image generated by the homography transformation has many problems, such as border pixels missing and misaligned textures with the original image, which greatly increase the residual information. In contrast, the disparity refers to the difference in coordinates of similar features within two stereo images, which can reflect the pixel-level correlation between the left and right images. As a 1D data flow, it is lightweight and can be easily compressed. Fig. 3 shows that the image generated by disparity-based warping is of better quality and has less residual information. Therefore, we propose to use the disparity map to assist with stereo image compression.
For convenience, we adopt an existing stereo matching model [59]. Taking the left image and the right image as input, it can generate the disparity map aligned with . On the decoder side, based on the decoded disparity map^, following [24] we can obtain the left predicted image^→ by using a bilinear sampler to sample the decoded right image^with backward mapping. The whole process is fully differentiable and can integrate into our end-to-end trainable compression network.

Aligned Cross-View Priors
Given the latent^, the entropy model^is used to fit the marginal distribution (^). The smallest average code length (bitrates) is decided by the matching degree of^and (^), given by the Shannon cross entropy between the two distributions: For the quantized hyper latent^,^and^, we use a nonparametric, fully factorized density model [3]. For the right quantized latent^and disparity map quantized latent^, we use an autoregressive context model with a mean and scale hyperprior [45], which can be formulated as: where , are the estimated mean and scale parameters.
Due to the large overlapping fields of view between the left and right cameras, strong correlation exists between the left quantized latent^and right quantized^. We visualize the features of the left and right images in Figure 4. It is obvious that^and^have structural similarities, so the mutual information (^,^) is positive. We denotes the Shannon entropy of^as (^), and then the conditional entropy (^|^) can be obtained: Since^is transmitted before^,^can serve as the cross-view priors to model the distribution^.
Moreover, considering that the disparity also exists betweenâ nd^, e.g. Figure 4 shows that the bag features in^have horizontal position offsets compared to^, directly feeding^as priors into the entropy estimation network of^like [16,35] is not accurate enough. So we warp the^based on the decoded disparity map^to get aligned latent codes^→ . Besides, we design a prior refinement net (PRN) (·) to refine^→ , removing the spatial discontinuity introduced by the warping operation. The generation process of the final aligned cross-view priors can be formulated by: ^)).
(4) Figure 5 illustrates the structure of our conditional entropy model. We use the hyperprior model [45] and the auto regressive network [45] to learn the hierarchical and spatial priors, respectively. Then we fuse these two priors with the aligned cross-view priors. We model the latent^as a conditional Gaussian distribution:^|^, where^represents the left quantized hyper latents, and are the estimated mean and scale parameters of left quantized latentŝ . The rate of the left image , the right image and the disparity map can be written as:

Training Strategy
Image compression aims to reconstruct the image of best quality with the least bitrates, which is a Lagrangian multiplier-based ratedistortion optimization problem. The loss function is formulated as: where denotes the bitrates, denotes the distortion and controls the rate-distortion trade-off.
To stabilize the training process, we use the pre-trained stereo matching model [59] as the initialization of disparity estimation. In addition, we introduce an auxiliary loss ( ,^→ ) at the early stage of training to instruct the whole network to utilize the disparity compensated prediction. The initial loss function L is defined as follows: where the hyperparameter is set to 0.2. When the training loss L is stable, the auxiliary loss ( ,^→ ) is removed and is set to 0. The total rate-distortion loss L can be formulated as:

EXPERIMENTS 4.1 Experimental Setup
Datasets. We evaluate the compression performance of our proposed DispSIC on two public stereo image datasets: (1) KITTI [19]: A dataset of outdoor scenes. The stereo images in it are with far views. We use 1,950, 50 and 50 stereo image pairs which are randomly selected by HESIC [16] for training, validation and testing respectively. During training, the images are randomly cropped to 256 × 512 patches. By using these two datasets of different scenarios, we can evaluate our method comprehensively. Implementation details. We implement our framework on the CompressAI PyTorch library [6]. We optimize our network using Adam optimizer [28] with a batch size of 4. We first train our model on KITTI, and then the model is fine-tuned to InStereo2K. All models are trained on a single V100 GPU for 1600 epochs. The learning rate is 1 × 10 −4 for 1200 epochs and reduced to 1 × 10 −5 for the last iterations. We train our network with the loss L in the first 800 epochs, and then with the loss L in the remaining epochs. We use mean square error (MSE) as our quality metric, and train 5 models for both datasets. The is chosen from set {0.001, 0.002, 0.005, 0.01, 0.02}.
Evaluation. To evaluate the rate-distortion performance, we compare our model DispSIC with the learned stereo image compression methods [16,35], the SOTA learned single image compression methods [12,23], and the traditional image and video compression codecs [10,53,54] on the KITTI and InStereo2K dataset. It is worth mentioning that when comparing with HEVC [54], stereo images are fed into the encoder as video sequences. The rate is measured by bits per pixel (bpp) and the quality is measured by peak signal-tonoise ratio (PSNR) and multi-scale structural similarity (MS-SSIM). Note the bpp is the average of both images. To compare the coding efficiency of different methods, we draw the rate-distortion (RD) curves. In addition, we report the Bjøntegaard delta PSNR (BD-PSNR) [9] and BD-rate results to better compare methods. The

Compression Results of Proposed Methods
Quantitative results. Figure 6 shows the rate-distortion performance of different methods on the KITTI and InStereo2K datasets. The data points on the RD curves are collected from the official GitHub page of HESIC [16]. For the MS-SSIM curve, in addition to the original scale, we also draw the log-scale, defined as −10 log 10 (1-MS-SSIM). Compared with the state-of-art (SOTA) stereo image Figure 7: Qualitative comparison. It is worth mentioning that these images are not cherry-picked, they are the same as images in HESIC [16].
compression method HESIC+ [16], the SOTA learned image compression method Cheng [12], and the SOTA traditional image compression codec VVC-intra [53], our method achieves the best performance on both datasets, which demonstrates the effectiveness and generality of our method. It is worth mentioning that our method achieves greater gain on the KITTI dataset at high bitrates. This demonstrates that our disparity-based method is more effective in scenes with complex structures. Table 1 shows the BD-BR and BD-PSNR results with DSIC [35] as the baseline, we can see that our method achieves the highest BD-PSNR and the lowest BD-rate.
Qualitative results. In Fig. 7, we provide some reconstructed images of different methods on the InStereo2K dataset. For a fair comparison, all images are compressed to similar bit rates. As we can see, our model achieves higher PSNR of reconstructed images at lower bitrates. Meanwhile, it can be seen that our method can generate more structural details than all other methods, such as the stripes on the tiger and the lines on the basketball. Computational complexity. In Table 2, we compare the FLOPs and parameters of DispSIC with HESIC+ and DSIC. Specifically, the resolution of the input image is 256 × 256. It can be seen that the Flops of DispSIC are slightly smaller than HESIC+ and much smaller than DSIC, and the number of parameters is much smaller than both HESIC+ and DSIC. The dense warp scheme of DSIC and the autoregressive model of HESIC+ for generating the H matrix have high computational complexity. Instead, we use a lightweight stereo matching model to explore the mutual information between stereo images, which greatly reduces the computational complexity. All in all, our model DispSIC has better performance and less complexity.

Analysis of Rate Allocation
In Table 4, we count the rate allocation of our method at different bitrates in the RD-curves on the KITTI dataset. Note the calculation of , , and is the bitstream size divided by the number of pixels in a single image, and → denotes the PSNR of the left disparity-based prediction.
As we can see, our network assigns the most bitrates ( ) to the right image. As to the left image, we compress the left residuals and the disparity map. The bitrates for the left residuals ( ) are less. Moreover, the disparity map is assigned the least bitrates ( ). Nonetheless, it can provide a respectable disparity compensated prediction, which greatly reduces the left residuals. In conclusion, by end-to-end optimization, our network can trade off the transmission cost and gain of the disparity map and adaptively allocate bitrates to the three parts to achieve the optimal compression performance.

Ablation Study
To analyze the contribution of each module, we implement ablation experiments on the KITTI and InStereo2K datasets as Figure 8 shows. Table 3 shows the specific configuration of ablation study.
Disparity-based Prediction. In case 1, we remove the disparitybased prediction and the priors. That means the stereo images are fed to the encoder separately, which can be considered as single image compression. As shown in Fig. 8, removing the disparity-based prediction causes significant performance degradation (case 1 vs. case 2), which demonstrates that the disparity-based prediction is the core module of our framework.
Aligned Cross-View Priors. To exploit the effectiveness of the aligned cross-view priors, we implement three experiments. In case 2, we simply remove the priors and use two independent entropy models for left and right images. In case 3, cross-view priors are sent to the left image entropy model, but lack alignment operation. In case 4, we align the cross-view priors with the left features. As we can see, the RD performance has a large drop if priors are disabled (case 2 vs. case 3). When enabling aligned cross-view priors (case 3 vs. case 4), the RD performance can be further improved, especially in scenes more complex. These two comparative experiments demonstrate the effectiveness of our aligned cross-view priors.
Prior Refinement Net. Since the warping operation may introduce some spatial discontinuity, we design a prior refinement net to further improve the quality of the aligned cross-view priors. As shown in Fig. 8, the RD curve of case 4 is lower than the original model DispSIC. This result shows that the prior refinement net is helpful to improve the quality of the priors.

CONCLUSION
In this paper, we propose a disparity-based stereo image compression network, which outperforms other existing SIC methods. Based on the disparity map, the right image is warped to the left view to get the residual image. Then the right image, the disparity map and the residual image are encoded into bitstream. Moreover, we propose a conditional entropy model with aligned cross-view priors, which provides more accurate probability estimation. Experiments on the KITTI and InStereo2K datasets show that our algorithm outperforms state-of-art learned stereo image compression methods.