{'fraction_non_alphanumeric': 0.02961568400282399, 'fraction_numerical': 0.0067160261400047066, 'mean_word_length': 5.111516760703617, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning End-to-End Lossy Image Compression: A Benchmark

Image compression is one of the most fundamental techniques and commonly used applications in the image and video processing field. Earlier methods built a well-designed pipeline, and efforts were made to improve all modules of the pipeline by handcrafted tuning. Later, tremendous contributions were made, especially when data-driven methods revitalized the domain with their excellent modeling capacities and flexibility in incorporating newly designed modules and constraints. Despite great progress, a systematic benchmark and comprehensive analysis of end-to-end learned image compression methods are lacking. In this paper, we first conduct a comprehensive literature survey of learned image compression methods. The literature is organized based on several aspects to jointly optimize the rate-distortion performance with a neural network, i.e., network architecture, entropy model and rate control. We describe milestones in cutting-edge learned image-compression methods, review a broad range of existing works, and provide insights into their historical development routes. With this survey, the main challenges of image compression methods are revealed, along with opportunities to address the related issues with recent advanced learning methods. This analysis provides an opportunity to take a further step towards higher-efficiency image compression. By introducing a coarse-to-fine hyperprior model for entropy estimation and signal reconstruction, we achieve improved rate-distortion performance, especially on high-resolution images. Extensive benchmark experiments demonstrate the superiority of our model in coding efficiency and the potential for acceleration by large-scale parallel computing devices.


INTRODUCTION
I Mage compression is a fundamental technique in the signal processing and computer vision fields. The constantly developing image and video compression methods facilitate the continual innovation of new applications, e.g. highresolution video streaming and augmented reality. The goal of image compression, especially lossy image compression, is to preserve the critical visual information of the image signal while reducing the bit-rate used to encode the image for efficient transmission and storage. For different application scenarios, trade-offs are made to balance the quality of the compressed image and the bit-rate of the code.
In recent decades, a variety of codecs have been developed to optimize the reconstruction quality with bitrate constraints. In the design of existing image compression frameworks, there are two basic principles. First, the image signal should be decorrelated, which is beneficial in improving the efficiency of entropy coding. Second, for lossy compression, the neglected information should have the least influence on the reconstruction quality, i.e., only the least important information for visual experience is discarded in the coding process.
The traditional transform image compression pipeline consists of several basic modules, i.e. transform, quantization and entropy coding. A well-designed transform for image compression transforms the image signal into compact and decorrelated coefficients. Discrete Cosine Trans- form (DCT) is applied to the 8 × 8 partitioned images in the JPEG [1] image coding standard. Discrete Wavelet Transform (DWT) in JPEG 2000 [2] further improves coding performance by introducing a multiresolution image representation to decorrelate images across scales. Then, quantization discards the least significant information by truncating less informative dimensions in the coefficient vectors. Methods are introduced to improve quantization performance, including vector quantization [3] and trellis-coded quantization [4]. After that, the decorrelated coefficients are compressed with entropy coding. Huffman coding is first employed in JPEG images. Then, improved entropy coding methods such as arithmetic coding [5] and context-adaptive binary arithmetic coding [6] are utilized in image and video codecs [7]. In addition to these basic components, modern video codecs, e.g. HEVC and VVC [8], employ intra prediction and an in-loop filter for intra-frame coding. These two components are also applied to BPG [9], an image codec, to further reduce spatial redundancy and improve the quality of the reconstruction frames, especially interblock redundancy. However, the widely used traditional hybrid image codecs have limitations. First, these methods are all based on partitioned blocks of images, which introduce blocking effects. Second, each module of the codec has a complex dependency on others. Thus, it is difficult to jointly optimize the whole codec. Third, as the model cannot be optimized as a whole, the partial improvement of one module may not bring a gain in overall performance, which makes it difficult to further improve the sophisticated framework.
With the rapid development of deep learning, there have been many works exploring the potential of artificial neural arXiv:2002.03711v2 [eess.IV] 19 Feb 2020 networks to form an end-to-end optimized image compression framework. The development of these learning-based methods has significant differences from traditional methods. For traditional methods, improved performance mainly comes from designing more complex tools for each component in the coding loop. Deeper analysis can be conducted on the input image, and more adaptive operations can be applied, which results in more compact codes. However, in some cases, although the performance of the single module is improved, the final performance of the codec, i.e., the superimposed performance of different modules, might not increase much, which makes further improvement difficult. For end-to-end learned methods, as the whole framework can be jointly optimized, performance improvement in a module naturally leads to a boost to the final objective. Furthermore, joint optimization causes all modules to work more adaptively with each other.
In the design of an end-to-end learned image compression method, two aspects are considered. First, if the latent representation coefficients after the transform network are less correlated, more bit-rate can be saved in the entropy coding. Second, if the probability distribution of the coefficients can be accurately estimated by an entropy model, the bit-stream can be utilized more efficiently and the bit-rate to encode the latent representations can be better controlled, thus, a better trade-off between the bitrate and the distortion can be achieved. The pioneering work of Toderici et al. [10] presents an end-to-end learned image compression that reconstructs the image by applying a recurrent neural network (RNN). Meanwhile, generalized divisive normalization (GDN) [11] was proposed by Ballé et al. to model image content with a density model, which shows an impressive capacity for image compression. Since that time, there have been numerous end-to-end learned image compression methods inspired by these frameworks.
Although tremendous progress has been made in endto-end learned image compression, there is a lack of a systematic survey and benchmark to summarize and compare different methods thoroughly. To this end, in this work, we conduct a comprehensive survey of recent progress in learning-based image compression as well as a thorough benchmarking analysis on different methods of learningbased image compression. The contributions and novelties of existing works are summarized and highlighted, and future directions are illustrated. With the summarized guidance from the survey and benchmark, we propose a novel end-to-end learned image compression framework that offers state-of-the-art performance.
The contributions of this paper are as follows: • We comprehensively summarize the existing end-toend learned image compression methods. The contributions and novelties of these methods are discussed and highlighted. The technical improvements of these methods are commented on based on their categorizations, which demonstrates a clear picture of the design methodologies and shows interesting future research directions.
• Inspired by the insights and challenges summarized for the existing approaches, we further explore the potential of end-to-end learned image compression and propose a coarse-to-fine hyperprior modeling framework for lossy image compression. The proposed method is shown to outperform existing methods in terms of not only the coding performance but also the potential to be accelerated.

•
We conduct a thorough benchmark analysis to compare the performance of existing end-to-end compression methods, the proposed method, and traditional codecs. The comparison is conducted fairly from different perspectives, i.e. the rate-distortion performance on different ranges of bit-rate or resolution and the complexity of the implementation.
Note that, this paper is the extension of our earlier publication [12]. We summarize the changes here. First, this paper additionally focuses on the thorough survey and benchmark of end-to-end learned image compression methods. In addition to [12], we summarize the contributions of existing works on end-to-end learned image compression in Sec. 3, and present a more detailed comparative analysis of the merits towards high-efficiency end-to-end learned image compression in Sec. 4 and Sec. 5. Second, we conduct a benchmark evaluation of existing methods in Sec. 7.2, where we present the comparative experimental results on two additional datasets, in both PSNR and MS-SSIM. Third, we raise the novel problem of cross-metric performance with respect to image compression methods in Sec. 7.3, where we present the empirical analysis on the phenomenon of crossmetric bias and we briefly discuss future research directions to address the related issues.
The rest of the paper is organized as follows. In Sec. 2, we first formulate the image compression problem, especially focusing on end-to-end learned schemes. After that, in Sec. 3, we briefly summarize the main contributions of existing research. Then, in Sec. 4, we categorize existing learned image compression methods according to their backbone models. After that, special attention is paid to the rate control technique in Sec. 5, which is the very specialized component in image compression compared with other deep-learning processing or understanding methods. Inspired by our survey and analysis, we introduce our new proposed method in Sec. 6. Later, in Sec. 7, we introduce the benchmarking protocols and make benchmarking comparisons of existing methods. Finally, in Sec. 8, we draw conclusions and discuss potential future research directions.

PROBLEM FORMULATION
Natural image signals include many spatial redundancies and have the potential to be compressed without much degradation in perceptual quality. Considering practical constraints on bandwidth and storage, lossy image compression is widely adopted to minimize the bit-rate of representing a given image to tolerate a certain level of distortion. The compression framework usually consists of an encoder-decoder pair. Given an input image x with its distribution p x , the encoder with an encoding transform E and a quantization function Q, a discrete code y is generated as follows: where θ E denotes the encoder parameters to be tuned for the learning-based image compression. To obtain the pixel representation of the image, the corresponding decoder D reconstructs the imagex from the code y as follows: where θ D denotes the parameters in D. Two kinds of metrics, i.e. distortion D and bit-rate R, give rise to rate-distortion optimization R + λD, the core problem of lossy image compression. The distortion term D measures how different the reconstructed image is from the original image, and it is usually measured via fidelity-driven metrics or perceptual metrics as follows: where d denotes the distortion function. The rate term R corresponds to the number of bits to encode y, which is bounded according to the entropy constraints. However, the actual probability distribution of the latent code y, denoted as p y , is unknown, making accurate entropy calculation intractable. Thus, we usually utilize an entropy model q y to serve as the estimation of p y for entropy coding. Hence, the rate term can be formulated as the cross entropy of p y and q y as follows: where p y stands for the real probability distribution and q y refers to the distribution estimated by the entropy model. The overall compression model can be viewed as an optimization of the weighted sum of R and D. Formally, the problem can be solved by minimizing the following optimization with a trade-off coefficient λ as follows: where θ p denotes the parameter for the entropy model. The optimal parametersθ E ,θ D ,θ p cause the model to achieve an overall good rate-distortion performance on the image x that follows x ∼ p x . Different λ values indicate different rate-distortion trade-offs, which depend on the requirements of different applications. Though the idea of rate-distortion optimization is also applied to traditional compression schemes, the joint optimization of all the components becomes feasible in learningbased methods. The opportunities and challenges are listed below: • Global Optimization. The major difference between learned image compression and the traditional hybrid codec lies in their optimizations. Instead of hand-craft tuning, the traditional hybrid codec can be tuned to any differentiable metric, e.g. SSIM [13], MS-SSIM [14] and perceptual difference [15], which is calculated by neural networks. In addition, while the traditional hybrid coding framework is usually improved at the scale of individual components, in learning-based methods, all modules are trainable, and it is possible to optimize all parameters and components jointly. However, it is nontrivial to acquire good performance in end-to-end learning compression because of the difficulties in optimization.
• Full-Resolution Processing. Convolutional neural networks support the full-resolution processing of images, while hybrid frameworks usually process partitioned blocks. Full processing can bring more benefits to entropy modeling with more context and avoid the blocking effect caused by partitioning. Fullresolution processing also comes with an increase in complexity. Because the perceptive field of a convolutional kernel is limited, the network needs to be deepened to perceive more large regions and improve modeling capacity.
• Rate Control. With joint optimization, the whole model can directly target the rate-distortion constraint, while in hybrid schemes, the additional ratecontrol component is employed and may not produce an optimal approximation. However, for a large portion of learning-based methods, multiple models need to be trained for different rate-distortion tradeoffs. The other single-model variable-bit-rate architectures are usually much more time-consuming. Therefore, the practical applications of these methods are limited.

OVERVIEW OF PROGRESS IN RECENT YEARS
Since the pioneering work of Toderici et al. [10] in 2015 exploited recurrent neural networks for learned image compression, much progress has been made. Benefiting from the strong modeling capacity of deep networks, the performance of learned image compression has exceeded that of JPEG to BPG (HEVC Intra), and the performance gap is widening further. The milestones of learned image compression are summarized in Table 1. Early works aim to search for possible architectures to apply transform coding with neural networks and propose end-to-end trainable solutions. Ballé et al. [11], [16], [20] proposes a learning-based framework with GDN nonlinearity embedded analysis and synthesis transforms for learned image compression, while Toderici et al. utilize recurrent models for variable-rate learned compression [10], [17].
To make the network end-to-end trainable, the quantization component, which is not differentiable based on the definition, should be designed carefully and approximated by a differentiable process. Some works replace the true quantization with additive uniform noise [20], [24] while others use direct rounding in forwarding and backpropagate the gradient of y = x. In addition, Agustsson et al. [18] proposes replacing direct scalar quantization with soft-to-hard vector quantization, which makes the quantization smoother. Dumas et al. [36] designs a model that additionally learns the quantization parameters.
When the compression network is trainable, the next issue is to efficiently reduce spatial redundancy in the image signal, where the transform is usually a critical part. Some take the form of a convolutional neural network (CNN), e.g. GDN [11], [16], [20] or residual block with enhanced nonlinearity [19]. Others resort to a recurrent neural network (RNN) to infer latent representations progressively, which forms a scalable coding framework [10]. In each iteration, the network largely squeezes out the unnecessary bits in the latent representations. Therefore, the final representations are compact. Variable rate image compression with recurrent neural networks ICLR 16 The first work to utilize a convolutional LSTM network for variable-bit-rate end-to-end learned image compression.
GDN Transform [16] End-to-end optimization of nonlinear transform codes for perceptual quality PCS 16 Introduces GDN, a trainable decorrelation nonlinear normalization, which shows a great capability for image compression.
Full-Resolution RNN [17] Full Resolution Image Compression with Recurrent Neural Networks CVPR 17 The first practical recurrent model for variable-bit-rate fullresolution image compression.
Soft-to-Hard Quantization [18] Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations

NIPS 17
Introduces vector quantization for learned compression and proposes using soft-to-hard annealing techniques to improve the performance of networks with quantization.
Compressive Autoencoder [19] Lossy Image Compression with Compressive Autoencoder ICLR 17 Residual network is first employed for CNN-based image compression models. A Laplace-smoothed histogram is used as the entropy model.

GDN Network [20]
End-to-End Optimized Image Compression ICLR 17 Introduces the multilayer nonpartitioning end-to-end architecture with GDN for image compression.

Inpainting
Based [21] Learning to Inpaint for Image Compression NIPS 17 Utilizes image inpainting techniques in a recurrent framework to improve compression performance.
Real-Time Adversarial [22] Real-Time Adaptive Image Compression ICML 17 The first method to adopt a multiscale framework with adversarial loss for learned real-time image compression. After the transform, the compact latent representations are further compressed via entropy coding, where frequently occurring patterns are represented with few bits and rarely occurring patterns with many bits. Earlier works incorporate elementwise independent entropy models to estimate the probability distribution of the latent representations [19], [20] and independently encode each element with an arithmetic coder. With these initial trials, later advanced methods explicitly estimate entropy with hyperpriors [24], [26], predictive models [25], [32], [34] or other learned para-metric models [17], [27], [28].
In addition to the abovementioned methods, which target signal fidelity with learned transform coding frameworks, there are emerging works utilizing generative adversarial networks (GANs) for extremely low-bit-rate image compression. According to research on the human visual system, human eyes are less sensitive to pixelwise distortion in images, especially for areas with complex texture. Therefore, generative models such as conditional GANs can be employed to synthesize areas with low-bit-rate representations as guidance, which can be utilized to design high-efficiency image codecs. Rippel et al. [22] first proposed utilizing the adversarial loss function in an end-to-end framework to improve visual quality. In later literature, Agustsson et al. [37], Tschannen et al. [30] and Santurkar et al. [38] improve the capacity of adversarial learning by introducing advanced generative networks to provide superior reconstruction quality with extremely low bit-rates.
In summary, the tremendous progress in learned image compression unveils the power of machine learning techniques. Nevertheless, there are still a large number of problems to investigate, which requires a systematic benchmark to illustrate critical areas where end-to-end learned frameworks for image compression can be further improved. In the following, we first analyze the important components (i.e., the backbone architecture and entropy model) in detail and then conduct the benchmark analysis on the methods according to various aspects.

BACKBONES FOR IMAGE COMPRESSION
A typical neural network backbone for image compression is built upon the architecture of the variational autoencoder (VAE) [39]. The architecture encodes images into vectors in a latent space, forming a compact representation. With dimensionality reduction and entropy constraints, the redundancy in the image is squeezed out by the compressive transform. There have been a variety of architectures for the backbone of the framework, which can be coarsely divided into two categories, namely, one-time feed-forward frameworks and multistage recurrent frameworks. Each component in a onetime feed-forward framework conducts the feed-forward operation only once in the encoding and decoding procedure. Usually, multiple models need to be trained to cover different ranges of bit-rates, as the encoder and decoder networks determine the rate-distortion trade-off. In contrast, in multistage recurrent frameworks, an encoding component of the network iteratively conducts compression on the original and residual signals, and the number of iterations controls the rate-distortion trade-off. Each iteration encodes a portion of the residual signal with a certain amount of bits. Such a model can conduct variable-bit-rate compression on its own. In the following, we introduce both types of architectures and conduct a comparison analysis on them.

One-Time Feed-Forward Frameworks
One-time feed-forward frameworks have been most widely adopted for end-to-end learned image compression. Basic variations of the architectures in the literature are illustrated in Fig. 1.
The first end-to-end learned image compression with a one-time feed-forward structure was proposed by Ballé et al. [16], where the analysis and synthesis transforms for encoding and decoding are made up of a single-layer GDN and inverse GDN (iGDN). This structure is then improved to support full-resolution processing, with stride convolution and the corresponding transposed convolution [20]. In later works, the hyperprior network [24] is introduced to extract the side information from the latent representation  produced by the analysis transform, and the side information can improve the entropy estimation of the latent code. In addition to the frameworks equipped with GDN, another kind of feed-forward network utilizing residual blocks is proposed by Theis et al. [19] and Mentzer et al. [27]. These networks stack multiple residual blocks in both the encoder and decoder, greatly expanding the depth of the model. With deeper networks, the encoder and decoder can embed more complex prior images, and they have more flexibility in modeling nonlinear transforms. In addition, some works adopt a multiscale structure [22], [31], which also extends the capacity of the network. It is reported that a more complex design of an architecture with GDN may bring further improvements in compression performance [40], [41], but not as significant as that of other contributions, such as a hyperprior. Unlike other computer vision tasks, e.g. image recognition, where a deeper network can usually bring extra gain in performance [42], [43], it does not result in significant improvements in performance to extend the architecture complexity for learned image compression. Although deeper architectures can provide more fidelity to model the prior of the images, they are harder to train than shallower networks, especially with a hard bottleneck in the pipeline. However, with sufficient capacity, due to the characteristics of this problem, an end-to-end optimization process may easily fall into local minima, and therefore, performance is not significantly improved with increased complexity.

Multistage Recurrent Frameworks
The basic architecture and the variations of multistage recurrent frameworks for image compression are illustrated in Fig. 2.
The vanilla multistage framework, as an illustration of the concept, progressively encodes the residue to compress the image. For an example of the simplified case, in the first stage, there is no reconstructed signal, so the residue is the original image itself. After the encoding and reconstruction, the residual image with respect to the reconstructed and original image is pushed into the network to conduct the second-stage compression. As at each stage the compression loses some of the information, the output of the second stage is the degraded signal of the true residue. The framework compresses the residue and the residue of the residue progressively to achieve better quality. To finally reconstruct the original image, bits of all the stages are needed to decode the multistage residue maps, which are added together to form the decoded image. This kind of reconstruction process corresponds to the Incremental structure in Fig. 2. The vanilla multistage framework adopts a stateless structure, where the analysis of different stages of the residue is conducted independently. It is difficult for the network to simultaneously compress the image and the residue of all steps. Therefore, in the first practical multistage structure [10], a stateful framework utilizing long short-term memory (LSTM) architectures [44] is introduced. LSTM maintains a state during sequential processing that propagates the features of the images to the following steps to facilitate the modeling of the multilevel residue. Fig. 2 shows the unrolled stateful structure. In each stage, the modules in the pipeline take the currently processed residue and the state from the previous stage as the input. The states are updated and propagated for processing in the next step. There have been studies on the aggregation of the output of each stage. Baig et. al. [21] present and analyze different kinds of aggregation schemes. The basic Incremental scheme adds the output of all stages together to form the final decoded images. The loss function of the Incremental scheme usually includes a term to encourage the output of each stage to approximate the residue of the previous stage. A different way to combine all the stages is to treat the multistage structure as a residual network to form the Skip-Connection scheme. There is only one term in the loss function for such a scheme to require that the sum of all the stages reconstructs the original image. Unlike the Incremental structure, there is no explicit arrangement of the residue in the Skip-Connection structure. The outputs of all stages contribute to the final reconstruction, each as a supplement of the reconstruction quality with respect to the others. In addition to these two kinds of schemes, Baig et. al. reported that with the Stateful-Propagation structure and the corresponding residual-to-image prediction, where each step produces a prediction of the original image rather than the residual signal, the network achieves the best performance. In such a stateful propagation scheme, it is important to propagate the states of the layers to the next step to construct a refined decoding image.

Comparative Analysis
Each of the two categories of backbone architectures has its own properties and corresponding pros and cons. The differences are mainly due to the choice between the onetime structure and the progressive structure. Here are some main differences.
• Recurrent models can naturally handle variable-rate compression, while for the feed-forward network, multiple instances of networks need to be trained to support a variable range of bit-rates.
• Feed-forward networks are comparatively shallower, and the path of back-propagation is much shorter. Training such a network can be easier. In contrast, training the recurrent models requires the backpropagation through time (BPTT) technique, which is more complicated.
• Weights are shared across different stages in the recurrent model; thus, the total number of parameters for a practical image codec may require less storage for the parameters compared with one-time feed-forward models. However, residual signals and image signals are different in nature, making the training of a recurrent model more challenging.
• It usually takes more time for recurrent models to encode and decode an image because the network is executed multiple times.
Despite the pros and cons, existing works report higher rate-distortion performance in one-time feed-forward architectures [25], [28]. However, variable-bit-rate compression is commonly required by applications, which becomes the major barrier for end-to-end learned image compression methods to be adopted by existing systems. More efforts are needed to investigate an efficient way to achieve variablerate compression for learning-based approaches.

ENTROPY MODELS
Entropy coding is an important component in an image compression framework. According to information theory [45], the bit-rate needed to encode the signal is bounded by the information entropy, which corresponds to the probability distribution of the symbols in representing the signal. Thus, the entropy coding component is embedded in the end-to-end learned image compression framework to estimate the probability distribution of the latent representations and apply constraints on the entropy to reduce the bit-rate.
There is a large amount of research on entropy models for learned image compression. A summary of solutions to the problem of entropy modeling is presented in Table 2, and we illustrate the typical structure of different variations in Fig. 3.
Ideal entropy coding requires precise estimation of the joint distribution of the elements in the latent representations, for each instance of the image. In earlier works, those elements are assumed to be independently distributed [19], [20] to simplify the design. However, even with optimized transforms, it is still difficult to eliminate the spatial redundancy in the latent maps of the images. Thus, a variety of entropy models are proposed to further reduce  [23], [29], [46], and (c) the binary context model [17].
It is worth noting that in traditional hybrid frameworks, improvements of the entropy model only affect entropy coding performance. For the learned method, as all the components are jointly optimized, a better designed entropy model not only produces a more precise estimate of the entropy but also changes the patterns produced by the analysis transform. As a consequence, the design of the entropy model should also take the structure of other components in the pipeline into consideration.
In summary, existing methods aim to provide a flexible transform and an accurate entropy model, all of which are neural network-based and end-to-end trainable. In addition The proposed network directly produces binary codes, which are transmitted as the bit-stream without entropy modeling [10]. Optional external entropy codecs, such as adaptive arithmetic coding [22], can be applied to the bit-stream to improve coding efficiency.

Masked
In addition to the binary code, the network also constructs a mask from the feature to indicate the length of the binary code [23], [29], [46]. The mask is usually transmitted together with the bit-stream. With rate-control, the overall performance can be further improved compared to the direct scheme.

Context-Model
The probability distribution of all the symbols to be encoded is estimated by the network with previously coded symbols [17] and spatially adjacent symbols [28]. The context model can more accurately estimate the probability so that the entropy coding can be conducted with more efficiency.

Statistical Histogram
The probability distribution is estimated by a histogram of the symbols [18]. A variation of this scheme is to use a Laplace-smoothed histogram for better generalization [19].

Piecewise Linear
The probability density function (PDF) is approximated by a parametric piecewise linear function during training [20]. Context Adaptive Binary Arithmetic Coding (CABAC) [6] is used to practically compress the latent codes.

Parametric Factorized
A function p(x i ) = f (x i , θ) with trainable parameters θ is modeled to estimate the probability of a symbol x i . These parameters reflect the distribution of latent code through the training set and can be generalized for all images [35], [47].

Gaussian
Networks based on VAE assume that the latent code follows an elementwise Gaussian distribution. The loss function includes a term of cross-entropy between the actual distribution and the estimated Gaussian distribution to control the bit-rate [24], [25], [34].

Masked Convolution
Masked convolution can be seen as a simplified version of PixelCNN for conditional probability modeling. Convolutional layers with 1 × 1 kernels are usually cascaded after masked convolution to extract additional features for probability estimation [25].

Offline
The latent code produced by a given encoder is analyzed in tiles offline by learning a dictionary, and the indexes are transmitted with lossless compression [26].

Hyperprior
The hyperprior, transmitted in the bit-stream, encodes the parameters, i.e. the mean and scale value, of a Gaussian entropy model [24] to estimate the likelihoods of the elements to be encoded. It greatly improves the accuracy of the entropy model and it can be combined with the context model for enhanced modeling [25], [34].
to the main goal of rate-distortion performance, several issues need to be addressed in the exploration. The model should be adaptive to different ranges of resolutions, bitrates, and distortions. Currently, when high-resolution capturing and displaying devices emerge, high-efficiency compression of high-resolution images is a constantly growing need. On the other hand, with the rapid development of large-scale parallel computing devices, e.g. GPU, models should also be designed to take advantage of parallel computing devices for higher efficiency. According to the above analysis, the one-time feed-forward frameworks with convolutional neural network-powered hyperprior structures have more potential to be scalable to a high-resolution and to support large-scale parallelism. With this idea in mind, we adopt a one-time feed-forward framework and achieve one step towards obtaining superior performance with a newly proposed coarse-to-fine hyperprior compression model.

Coarse-to-Fine Hyperprior Modeling
As analyzed, we follow the basic framework of a onetime feed-forward framework, which consists of an analysis transform G a and a synthesis transform G s . G a transforms the image to latent representations, and G s reconstructs the image from those representations. To perform entropy coding, the latent representations are first quantized to a vector of discrete symbols X = {X 1 , X 2 , ..., X n }. In addition, a parametric entropy model Q X (X; θ) w.r.t the random vector X is built to provide the estimation of the likelihoods. The aim of entropy coding is now to jointly optimize the parameters in the networks to 1) accurately model the distribution P X (X) of the random vector X with Q X (X; θ) and 2) minimize the overall rate-distortion function with the estimated entropy. State-of-the-art methods combine context models and hyperpriors. In such approaches, it is first assumed that the joint probability distribution of X can be factorized to the product of sequential conditional probabilities as follows: where Y denotes the hyperprior, which is generated from X and encoded to the bit-stream. When we need to decode X, Y has already been decoded. These kinds of models need to address two issues. First, the dimensionality and the corresponding bit-rate of Y should be kept low; otherwise, Y itself may contain too much redundancy and is not efficiently compressed. In such a circumstance, the hyperprior may not provide enough information to accurately model the conditional probability, especially for higher ranges of bit-rates and large resolutions. Second, although contextual conditioning can help with accuracy, it is performed in a sequential way and is hard to accelerate with large-scale parallel computing devices. Thus, the framework is less scalable for input images of different sizes.
To address the issues of the sequential context models, in the proposed method, we adopt a multilayer conditioning framework, which improves scalability for images of different sizes. The formulation is modified as follows: The first equality in Eq. (7) holds for Y because the hyperprior is generated from X in a deterministic manner. When X becomes complex and is controlled by expanding the dimension, Y may need to embed more information to support accurate conditional modeling. Therefore, an additional layer of the hyperprior is introduced as follows: which in fact forms a coarse-to-fine hyperprior model. The dimension of Z is reduced, and the redundancy is squeezed out by the hypertransforms. Thus, the joint distribution P Z (Z) of the latent representation Z = {Z 1 , Z 2 , · · ·, Z n } at the innermost layer can be approximately factorized as follows: With Eq. (7) and Eq. (8), the probability distribution of Y and X can now be modeled in a conditional way, while existing works [48], [50] show that neural networks are capable of modeling conditional probability distributions. The hyperrepresentation Y is also designed to embed the main information of the images to be compressed. Therefore, the joint distribution can also be approximately factorized as follows: where all elements in the previous layer can be utilized as the conditions to estimate the distribution of the latent representation at the upper layer. Although no contextual conditioning is conducted here, contextual conditioning can be implicitly modeled in the information flow from X to Y and then used to predict X from Y. Unlike existing blockconditioning context models, in the proposed framework, the estimation of the probability for each element utilizes information from a larger area due to the coarse-to-fine structure. This helps to explore long-term correlations in images and improves the compression performance, especially for high-resolution images.

Network Architecture
The overall structure of the end-to-end learned coarse-tofine framework is shown in Fig. 4 jointly with the encoder and decoder. The analysis transform network encodes the input image as the latent representation X, which is then quantized with a rounding operation. It aims to squeeze out pixelwise redundancy as much as possible. We exploit GDN as the activation in the analysis transform and inverse GDN in the synthesis transform. We conduct coarse-to-fine modeling with multilayer hyper analysis and a symmetric hyper synthesis transform. According to Eq. (7) and Eq. (8), to estimate the distribution of X, a probability estimation network is employed to process Y and predict the likelihood P Xi (X i = x i ) with the estimated Q Xi (X i = x i ) for each element X i in X. As stated in [24], the conditional distribution of each element in X can be assumed to be Gaussian, and the probability estimation network predicts the mean and scale of the Gaussian distribution. As the latent code has been rounded to be discrete, the likelihood of the latent code can be calculated as follows: where φ denotes the cumulative distribution function of a standard normal distribution, while the mean µ xi and scale σ xi are predicted from Y. The same process is conducted w.r.t. Y and Z to estimate the probability distribution of Y.
As illustrated in Eq. (9), the probability distribution of Z can be approximately factorized. Thus, we employ a zero-mean Gaussian model. The likelihood of each element in Z can be calculated as follows: Note that σ i is a trainable parameter in the network. All channels in the latent representation share the same σ while each channel has an independent one. According to information theory, the minimum bit-rate required to encode X (or Y and Z) with the estimated distribution equals the cross entropy of the real distribution P X|Y (X|Y) and the estimated distribution Q X|Y (X|Y) ∼ N (µ x , σ x ), which is denoted as follows: We minimize the rate-distortion function L RD = R + λD with the network. To accelerate the convergence during the training of the multilayer network, an additional information-fidelity loss is introduced. This loss term encourages the hyperrepresentation Y to maintain the critical information in X during training and is formulated as follows: min In practice, the function F with trainable parameter θ is one convolutional layer with no nonlinear activation. The information-fidelity loss takes the form of the least-square error to make the prediction of µ and σ more accurate.

Signal-Preserving Hyper Transform
To conduct coarse-to-fine modeling of images, especially for high-fidelity modeling in high-resolution or high-quality circumstances, it is important to preserve the information while performing hyper analysis and synthesis transforms in the succeeding hyperlayers. Therefore, the signalpreserving hypertransform is proposed to build a framework with multiple layers. We observe that elements in the latent representations produced by the main analysis transform are much less correlated compared with pixels in natural images, as the spatial redundancy has been largely reduced by the previous analysis transforms. Therefore, local correlations in the feature maps are weak, while convolutions with large kernels rely on such local correlations for effective modeling. In addition, the previous transform network consists of stride convolutions with ReLU activation. Stride convolutions downsample the feature maps, while activation functions such as ReLU intuitively disable some of the filter neurons that produce negative values and make the response sparser. Because the dimension of these convolution layers needs to be limited to ensure the gradual factorization of the latent representation, the original hypertransform loses much information during processing.
In summary, the issues of the original analysis transform in the proposed architecture fall into two categories: 1) Original analysis transforms fix the number of channels and downsample the feature maps, which reduces the dimension of the latent maps. 2) Combining large convolution kernels with ReLUs at the beginning of the analysis transform or the end of the synthesis transform will lose (a) Hyper analysis transform.

Name
Operation Output Shape Activation some information that has not been transformed, limiting the capacity. The signal-preserving hypertransform is designed to facilitate the multilayer structure by preserving information for coarse-to-fine analysis. The structure of the analysis and synthesis transform network is illustrated in Table 3. Instead of using large kernels in the filters, we employ a relatively small filter in the first layer with no nonlinear activation, and we conduct 1 × 1 convolutions in the remaining layers. The first layer in the network expands the dimension of the original representations. Combined with succeeding nonlinear layers, the expansion of dimension preserves the information of the original representations while supporting nonlinear modeling. We exploit a space-todepth operation to reshape the tensor of the representations, making spatially adjacent elements scatter in one location but in different channels. In this way, the succeeding 1 × 1 convolutions are able to conduct nonlinear transforms to reduce spatial redundancy. At the final layer of the network, we conduct a dimensionality reduction on the tensor to make the representation compact. We symmetrically design the hyper synthesis transform to produce Y in Eq. (7) as the conditional prior for the outer layer, which is taken as the side information for reconstruction.

Information Aggregation for Reconstruction
In the decoding process, the synthesis transform maps latent representations back to pixels. To best reconstruct the image, the decoder needs to fully utilize the provided information in the bit stream. Practical image and video compression usually exploit side information to improve quality. With this idea in mind, we take hyperlatent representations as side information and aggregate information from all layers of the hyperlatent representations to reconstruct the decoded image in the proposed framework. The architecture of the information aggregation decoding network is shown in Fig. 5. Both the main latent representation and the higher order representations of smaller scales are upsampled by the decoding network to half the size of the output image. A fusion is conducted with a concatenation of the two representations. The fused representation is then processed by a residue block and then upsampled to the scale of the output image.   By fusing the main representation and the hyperrepresentations, information of different scales contributes to the reconstruction of the decoded image, where the higherorder representations provide global information and the others preserve details in the image. The fusion process is conducted at smaller spatial resolutions to avoid high computational complexity. After the fusion of features, we employ a single residue block with peripheral convolution layers to map the feature maps back to pixels.

Datasets
End-to-end learned image compression is a self-supervised problem where distortion metrics measure the difference between the original image and the reconstructed image and the bit-rate corresponds to the entropy of the latent code. Thus, no extra labeling labor is needed, and many existing large-scale image sets, e.g. ImageNet [51] and DIV2K [52], can be used to train networks for image compression. To reduce possible compression artifacts in the images, the lossy-compressed images are usually downsampled before they are used for network training.
Commonly used testing image sets include Kodak [53] and Tecnick [54], which contain high-quality natural images that have not been lossy-compressed. The Kodak dataset consists of 24 images with resolution 512 × 768, with a wide variety of content and textures that are sensitive to artifacts. Thus, it has been widely used to evaluate image compression methods. For the Tecnick dataset, the SAM-PLING testset is used for evaluation in some works. In contrast to Kodak, this dataset contains images with higher resolution (1200 × 1200), which can serve as a supplemental benchmark for image compression methods that can have different performance on images with different resolutions.
In addition, in recent years, the CVPR Workshop and Challenge on Learned Image Compression (CLIC), with the goal of encouraging research in learning-based image compression, has attracted much attention in the community. A testing dataset consisting of images captured by both mobile phones and professional cameras is provided and updated year by year. The images have higher resolutions, on average 1913 × 1361 for mobile photos and 1803 × 1175 for professional photos. Evaluation results on this dataset indicate compression performance on images with relatively high resolutions.

Rate-Distortion Performance
Although the overall history of the development of endto-end learned image-compression methods is not as long as that of hybrid coding standards, there have been a significant number of works on this topic, and tremendous progress has been made. However, few studies have thoroughly evaluated rate-distortion performance on various images and compared baselines (i.e., anchors). It is nevertheless valuable to compare performance on technical merits to investigate which direction truly affects performance. In the following, we summarize the performance of selected works. The contributions in these works include different methods for entropy modeling, novel architecture design and normalization.

Evaluation Protocol
Three datasets, i.e., Kodak, Tecnick and CLIC 19, are used in the evaluation, which correspond to three different levels of resolution and different content. For the evaluated learningbased methods, we average the metrics of the bit-rate (bpp) and the distortion (PSNR and MS-SSIM) across the dataset for different models, which are usually trained with different trade-off coefficients λ. All methods are compared with JPEG and BPG 4:4:4. For both JPEG and BPG, the metrics are averaged at different quality factors (QFs) or quantization parameters (QPs). To illustrate the comparison, we show the results for rate-distortion curves in Fig. 6. We also calculate the BD-rate [55] with respect to the bit-rate and PSNR over the three datasets, as illustrated in Table 4  and that all the methods can be included in the comparison. We analyze the results and summarize the important properties in the following.

Entropy Model
The design of the entropy model is the main driving force of improvements in rate-distortion performance. The design of entropy models in end-to-end learned image compression has developed through the period from contextual binary entropy models [17] to hyperprior models and spatial / cross-channel entropy estimation [24], [25]. Specifically, as shown in Fig. 6, a leap in gain occurred with the emergence of hyperpriors, which have been adopted by many other frameworks. Despite great success, modeling contextual probability is still a challenging topic in image modeling due to variation in resolution. As shown in Table 4, context model-based methods [25], [34] may have unstable gain over BPG at different levels of resolution, while the proposed methods achieve consistent superiority over the anchor.

Depth of the Network
The depth of the network is a comparatively less important factor in performance, while in other computer vision tasks, networks with a deeper architecture usually perform better than those with fewer layers. Some works [40], [56] also confirm this observation. Instead of building complicated network architectures, work may focus more on the specific design of the networks to better model the image prior. However, it has been reported that the network should consist of a sufficient quantity of parameters, and the width of the network should be large enough for effective modeling of images, especially for higher ranges of bit-rates and higher quality [34].

Normalization
It is reported in [24] that batch normalization [57], commonly used to improve the performance of neural networks, does not bring significant improvement. However, Ballè et al. proposed generalized divisive normalization [11], [47], which is proven to be able to decorrelate the elements in images to improve overall performance. Most state-of-theart solutions adopt normalization and its inverse in the main encoding and decoding transform. However, it still remains as a topic in future research to reduce spatial redundancy more efficiently with normalization.

Summary
We evaluate the rate-distortion performance of different methods developed in recent years. As we can see from the results, great progress has been made to improve the rate-distortion performance, where the decorrelation normalization and the hyperprior model bring significant improvement. Nevertheless, we also see large variations in performance on different testing datasets. Compared with existing works, the proposed method achieves a more consistent gain on different content and resolutions.

Cross-Metric Evaluation
End-to-end learned image compression models can usually be trained towards many objectives as long as they are differentiable. Because end-to-end learned models can be tuned with both PSNR and MS-SSIM, we are able to investigate the relationship between different metrics and objectives. We employ the perceptual metric, which is widely used in image enhancement and synthesis [15], [58], as the metric for cross evaluation. Following the settings of perceptual loss in [15], the L 2 distance of the output feature maps corresponding to four layers in the VGG-16 [59] with respect to the original image and the reconstructed image are evaluated. Zhang et al. [58] show that the distance of the feature maps of such layers reflects the distortion with respect to human perception. Thus, we employ the metric as a supporting evaluation of the reconstruction quality for image compression methods.
To show the comparison, we plot the Perceptual-PSNR and Perceptual-MS-SSIM curves in Fig. 7. Note that the distances with respect to the four layers are averaged for the illustration. Here are our observations of the experimental results.
• For a given level of PSNR, models trained on MS-SSIM show significantly less perceptual distortion, while for a given level of MS-SSIM, those trained on PSNR have less perceptual distortion. When a model is optimized for a metric, compared with others that are not optimized for that metric, the optimized one    shows a higher perceptual distortion at the same level of the metric.
• For models tuned with the MS-SSIM loss function, those with higher performance in MS-SSIMbpp evaluation tend to result in larger perceptual distortion at a certain level of MS-SSIM. Although the same phenomenon is observed in the Perceptual-PSNR curve, it is not as significant.
To summarize, we observe in the experimental results that there exists a gap of different metrics, especially for models with better performance on one metric. Although end-to-end learning-based methods can be trained towards different objectives, they tend to be over-optimized on that specific objective only. This phenomenon is also related to recent work on the investigation into the trade-off between perception and distortion [60], [61]. In circumstances where we reserve more bit-rate for an image encoded with a better codec, as metrics such as PSNR and MS-SSIM show high enough values at that bit-rate, we may not be provided with the expected visual quality. A better assessment technique is needed, especially for the development of high-performance image compression methods. Furthermore, in real-world applications, the images are mostly consumed by human users, while there is a trend of developing image processing systems for machine vision tasks. To jointly optimize an image compression framework for both human perception and machine intelligence remains to be explored in future research.