{'fraction_non_alphanumeric': 0.03400932107318302, 'fraction_numerical': 0.01263803165805937, 'mean_word_length': 4.994966020639315, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CNeRV: Content-adaptive Neural Representation for Visual Data

Compression and reconstruction of visual data have been widely studied in the computer vision community, even before the popularization of deep learning. More recently, some have used deep learning to improve or refine existing pipelines, while others have proposed end-to-end approaches, including autoencoders and implicit neural representations, such as SIREN and NeRV. In this work, we propose Neural Visual Representation with Content-adaptive Embedding (CNeRV), which combines the generalizability of autoencoders with the simplicity and compactness of implicit representation. We introduce a novel content-adaptive embedding that is unified, concise, and internally (within-video) generalizable, that compliments a powerful decoder with a single-layer encoder. We match the performance of NeRV, a state-of-the-art implicit neural representation, on the reconstruction task for frames seen during training while far surpassing for frames that are skipped during training (unseen images). To achieve similar reconstruction quality on unseen images, NeRV needs 120x more time to overfit per-frame due to its lack of internal generalization. With the same latent code length and similar model size, CNeRV outperforms autoencoders on reconstruction of both seen and unseen images. We also show promising results for visual data compression. More details can be found in the project pagehttps://haochen-rye.github.io/CNeRV/


Introduction
Visual data compression remains a fundamental problem in computer vision, and most methods can be seen as autoencoders, consisting of two components: encoder and decoder. Traditional compression methods, such as JPEG [46], H.264 [48], and HEVC [40], manually design the encoder and decoder based on discrete cosine transform (DCT) [3]. With the success of deep learning, many attempts [2,8,15,21,22,24,34,35,49] have been made to replace certain components of existing compression pipelines with neural networks. Although these learning-based compression methods show high potential in terms of rate-distortion performance, they suffer from expensive computation, not just to train, but also to encode and decode. Moreover, as a result of being partially hand-crafted, they are also quite reliant on various hard-coded priors.
To address the heavy computation for autoencoders, implicit neural representations [7,30,33,36,38] have become popular due to simplicity, compactness, and efficiency. These methods show great potential for visual data compression, such as COIN [13] for image compression, and NeRV [5] for video compression. By representing visual data as neural networks, visual data compression problems can be converted to model compression problems and greatly simplify the complex encoding and decoding pipeline.
Unlike other implicit methods that map a single network to a single image, NeRV trains as a single network to map timestamps to RGB frames directly for entire videos. This allows NeRV to achieve incredible results for video compression. However, because NeRV's input embedding comes from the positional encoding of an image/frame index, which is contentagnostic, NeRV can only memorize. This is evidenced by it achieving surprisingly poor reconstruction quality for unseen data (images/frames that are skipped during training), even when these images only deviate very slightly from images it has seen. This means that NeRV can only work with a fixed set of images that it has seen during training time, and it could never perform, for example, post-training operations such as frame interpolation.
We thus propose Content-adaptive Neural Representation for Visual Data (CNeRV) to enable internal generalization. With a content-adaptive embedding, rather than a temporal/indexbased embedding, CNeRV combines the generalizability of autoencoders (AEs) with the simplicity and compactness of implicit representation. Similar to implicit representations, CNeRV has a strong decoder, and stores most of the visual prior in the neural network itself. Given a tiny embedding, CNeRV can reconstruct the image with high quality, just as NeRV does, and serves as an internally-generalizable neural representation, shown in Figure 1.
We summarize our primary contributions as follows: • We propose content-adaptive embedding (CAE) to effectively and compactly encode visual information, and generalize to skipped images for a given video/domain. • We propose CNeRV based on CAE, which leverages a single-layer mini-network to encode images (120× faster than NeRV), with no need for the time-consuming perimage overfitting used by implicit representation methods.
• We demonstrate that CNeRV outperforms autoencoders on the reconstruction task (+9.5db for seen image PSNR, +3.5db for unseen image PSNR).
• We show promising video compression results for both unseen and all frames when compared with traditional visual codec such as H.264 and HEVC.

Related Work
Neural Representation. Implicit neural representations can be divided into two types: pixel-wise representation and image-wise representation. Taking pixel coordinate as input, pixel-wise representation yields outputs based on the input queries, and have become popular for numerous applications, including image reconstruction [38], shape regression [7,30], and 3D view synthesis [36]. For image-wise implicit representation, NeRV [5] outputs the whole image given an index, which greatly speeds up the encoding and decoding process compared to pixel-wise methods, and a recent E-NeRV [20] improves the architecture design. CNeRV is also an image-wise representation method.  [41].
Autoencoders. Our work is related to other works where a network learns to represent an image in a way that either relies on or later lends itself to reconstruction of the image. Of these methods, ours is closely related to auto-encoding [3,17,32,45], which focuses on encoding and reconstruction of real images, sometimes by leveraging adversarial techniques [9,10,11,25] that were originally proposed to help synthesize new images [14]. Numerous algorithmic and architectural improvements [6,12,43,44] were introduced later based on the vanilla autoencoder. We take vanilla convolutional autoencoder and convolutional VAE as baselines in this work. In fact, our method is akin to an autoencoder that is optimized for data compression: the encoder is very small and fast, while the decoder is reasonably quick and not excessively large.
Visual Codec. Borrowing principles from image compression techniques [39,46] and transform coding methods [3,4], traditional video compression methods such as MPEG [18], H.264 [48], and HEVC [40] are designed to be both fast and accurate. Recently, deep learning techniques have been proposed to replace portions of the video compression pipeline [2,5,8,15,16,21,22,23,34,35,49]. Although these learning-based methods show promising results for rate-distortion performance, most of them suffer from complex pipelines and heavy computation. Of all the methods referenced in this section, ours is most closely related NeRV [5], which converts frame index to a positional encoding to allow a neural network to memorize and compress a video. Covered in more detail in Sec. 3, a primary difference from NeRV is CNeRV's use of a content-adaptive embedding, which allows CNeRV to encode frames that were skipped during training time.

Method
Our work on representation and compression for visual data builds on NeRV. We replace their content-agnostic positional embedding with a proposed content adaptive embedding and a single-layer neural encoder. To clarify the relationship between CNeRV and NeRV, we first revisit NeRV in Sec. 3.1, then present CNeRV to introduce internal generalization in Sec. 3.2, and finally how it can be leveraged for visual data compression in Sec. 3.3.

Revisiting NeRV
As shown in Figure 2, NeRV takes as input an image index t, normalized between 0 and 1, and outputs the whole image directly, through an embedding layer and a neural network. The image embedding is given by a positional encoding function: where t is the image index, b is the frequency value, and l is the frequency length. Specifically, the NeRV network, as illustrated in Figure 3(a), consists of a multi-layer perceptron (MLP) and stacked NeRV blocks. To upscale the spatial size, a NeRV block stacks a convolution layer, a pixelshuffle module [37], and an activation layer, as illustrated in Figure 3(b).

CNeRV: Content-adaptive Neural Representation for Visual Data
To ameliorate NeRV's lack of internal generalization (results in supplementary material), motivated by its failure to properly correlate neighboring frames, we propose a different embedding. Rather than a positional encoding, which is content-agnostic, we design a contentaware encoding. We create a single-layer encoder to perform a learned transform on this encoding and use NeRV decoder for reconstruction. CNeRV Encoder We introduce a tiny, single-layer encoder in Figure 2(c) that uses contentadaptive embedding to achieve internal generalization. Dividing image into blocks. One insight behind CNeRV is that local visual patterns can be extracted and act as a visual prior. To achieve this, we first divide each input image into blocks and encode them in a batch, illustrated in step 1 at Figure 2 where R C×H×W represents the input image, C the image channel, H and W image height and width, M and N block numbers Content-adaptive embedding. To improve the embedding generalization, we encode the image content into an embedding. Inspired by DCT [3] and positional encoding, we compute content-adaptive embedding R C×P×Q with where b is the frequency value, p and q are the frequency values, Img(c, x, y) is the pixel value at location (x, y) for channel c, and x and y are normalized between (0, 1). We then concatenate the block embedding to construct the image embedding R CPQ×M×N as shown in Figure 2(c) with steps 2 and 3. Reduce embedding dimension. Since the raw embedding can be very high in length dimension (CPQ), we introduce a 1 × 1 convolution to reduce the length and get an embedding R L×M×N . This will be the image latent code and also the input of CNeRV decoder, as illustrated in Figure 2(c), step 4. CNeRV Decoder We illustrate the CNeRV decoder in Figure 2(d) and divide it into two parts: block-wise and image-wise decoding.
Block-wise Decoding. Given image embedding R L×M×N , we firstly decode it blockwisely with 1 × 1 convolution into R dhw×M×N . Then we reshape it into R d×Mh×Nw , which means every block embedding R L×1×1 is decoded into a R d×h×w cube. Since this is shared by blocks, we refer to it as block-wise decoding.
Image-wise Decoding. Given R d×Mh×Nw , following NeRV [5], we upscale the feature map into the final image output R C×H×W with stacked NeRV blocks. Since NeRV block consists of 3 × 3 convolution and fuses information over the whole image, we therefore refer to it as image-wise decoding. Loss Objective Following NeRV [5], we adopt a combination of L1 and SSIM loss as our loss function for network optimization, following where y and v are the ground truth image and CNeRV image prediction, and α is a hyperparameter to balance the loss items.

Application: Visual Data Compression
Following NeRV, we use model pruning, quantization, and entropy encoding for model compression. Similar to model quantization, we also apply embedding quantization for visual data compression. Given a tensor µ, a tensor element µ i at position i is 'Round' is a function that rounds to the closest integer, 'bit' is the bit length for quantization, µ max and µ min are the max and min value for µ, and 'scale' is the scaling factor. Using this equation, each model parameter or frame embedding value can be represented with only 'bit' length -by compressing the model and embedding in this way, we achieve visual data compression. When computing quantized model size, we also use entropy encoding as an off-the-shelf technique to save space.       original paper. We evaluate the video quality with two metrics: PSNR and MS-SSIM. Bitsper-pixel (BPP) is adopted to evaluate the compression ratio. We implement our method in PyTorch [31] and train it in full precision (FP32), on NVIDIA RTX2080Ti.
We firstly fit the model on the seen split, and evaluate its internal generalization on the unseen set. When computing "total size" for a representation method, for implicit representation (e.g., NeRV) we only compute model parameters, while for methods with image latent code, we compute both model parameters and image embedding size as total size. More results can be found in the supplementary material. Comparison to NeRV We present our main result in Table 2, which shows that CNeRV consistently outperforms NeRV in terms of both reconstruction quality of unseen images (unseen PSNR) and internal generalizability (PSNR gap between seen and unseen frames/images). Extended tables with more details and results in this section can be found in the supplementary material. Note that the generalization of NeRV becomes worse when FPS decreases, i.e., with diversified frame content, while CNeRV retains its generalizability in all cases. We first verify CNeRV's superior performance holds under a variety of settings, increasing model size in Table 3, various video resolution in Table 4, various training images in Table 5.
We then extend our finding on the non-sequential nature of NeRV's representation to apply it to an image dataset, using an arbitrary index for each image as the frame index. NeRV thus takes image index (1/N, ..., N/N respectively for N images) as input. With this adaptation, we compare them in Table 6 and CNeRV shows superior internal generalization.
There is no question that NeRV and CNeRV both succeed by fitting to their training data. However, as Figure 4 shows, CNeRV is able to do this without sacrificing generalization for increasing training epochs (it does not exhibit the same overfitting behavior). CNeRV's performance for unseen images doesn't decrease as training time increases, even though its seen        Table 8 shows reconstruction results for both seen and unseen images, as well as encoder model size and encoding speed. Note that for encoding time, we only consider the forward time, ignoring the data loading overhead. With a tiny encoder and strong decoder, CNeRV outperforms autoencoders and pixel-wise neural representations in many regards, including reconstruction quality of seen and unseen images, and encoding speed. We show visualization results for seen images in Figure 11 and unseen images in Figure 12. With the same embedding length and similar total size, CNeRV outperforms NeRV with better detail, absence of artifacts, and lack of spillover from previous frames. Although autoencoder with large embeddings can reach comparable generalization for unseen images, it struggles a lot for reconstruction of seen images. But, visual differences still exist between CNeRV and ground truth, and future work can focus on mitigating these issues.
For low reconstruction quality of pixel-wise neural representations, we believe both these methods are designed and optimized for low-resolution images, and a similar perfor- mance drop on high-resolution images is also observed in MLF [26]. For the ConvAE and ConvVAE, we speculate two potential causes for the inferior reconstruction capacity. First, due to the fact that the parameters are more evenly balanced between encoder and decoder, they rely on larger embeddings and struggle for our embedding size. Second, we speculate that small training datasets likely limit their capability as well. We show in the supplementary material that increasing model size, embedding length, and amount of training data each make autoencoders more competitive with CNeRV.
We also compare decoding speed with traditional codecs in Table 9. The decoding of traditional codecs are measured with 8 CPUs, while CNeRV is measured on 1 RTX2080ti. As a video neural representation, CNeRV shows good decoding advantage due to its simplicity and can be deployed easily. Frame interpolation and Visualization Given neighboring frames are typically similar, we investigate whether using CNeRV to encode unseen frames is better than interpolating from the embeddings of the neighboring seen frames. We show these results in Table 10. Interpolated embeddings achieve similar performance with actual CNeRV embeddings for unseen images across the lower FPS datasets, but significantly lower results for the Bunny dataset. With the same embedding length and similar total size, CNeRV outperforms NeRV and autoencoder with better detail for both seen and unseen frames. Visual Data Compression We also show visual data compression results for CNeRV, as discussed in Sec. 3.3. We compare the rate-distortion results with traditional visual codec such as H.264 and HEVC. For unseen frames, we compare visual comparison results in Figure 7 where CNeRV bitrates only consider image embedding as all other autoencoder methods do [2,8,23,35]. Besides, we evaluate compression results on the full video (both seen and unseen frames) frames where we combine both the image embedding and model parameters to compute bitrates, CNeRV outperforms H.264 and HEVC on both PSNR and SSIM with similar bpp in Figure 8.

Conclusion
In this work, we propose a content-adaptive neural representation, CNeRV. CNeRV combines the generalizability of autoencoders and simplicity and compactness of neural representation. With a single-layer mini-encoder to generate the embedding, CNeRV outperforms autoencoders for the reconstruction task in terms of image quality (+3.5db for unseen image PSNR), encoding speed (36× faster), and encoder size (116× smaller). We leverage this content-adaptive embedding with CNeRV to encode unseen images quickly (120× faster than NeRV), with no need for the time-consuming per-image overfitting. We also show promising visual data compression results and provide embedding analysis.
Acknowledgement. This project was partially funded by an independent grant from Facebook AI and the DARPA SAIL-ON (W911NF2020009) program.

A.1 Embedding Quantization Results
For unseen frames and with an 8-bit quantized model, we further quantize the block embedding, ranging from the original 32 bit to 1 bit, where where the image embedding can maintain most of its capacity with only 6 bits in Figure 9

A.2 More Autoencoder Results
We first provide more autoencoder results, where both seen PSNR and unseen PSNR improve as we increase the training images, embedding length, and model size.   We conduct an ablation study on our content-adaptive encoding. For base value b in Equation 2, Table 12 shows that 1.15 performs better than 1.05 and 1.25. For frequency length P and Q, Table 13 shows results with 10, 15, and 20. Although 20 is better for seen PSNR, it does not further improve unseen PSNR and will introduce more encoding computation. Since we mainly focus on internal generalizability in this work, i.e., unseen image PSNR, we choose 15 as the default frequency length. For block numbers, Table 10 shows results for 1 × 2, 2 × 4, and 5 × 10. where total embedding length is computed by M × N × L. With 2 × 4 blocks, CNeRV reaches the best performance for both seen and unseen images. Note that when embedding length is too short (e.g., 10 for block number 5 × 10), CNeRV fails to overfit; nevertheless, it can still perform reasonable reconstruction. Limitation The main limitation of our method is that with a small training dataset, unseen PSNR still lags behind seen PSNR. Although more training images can alleviate this prob-  Figure 11: Video compression results for seen frames from video Bunny. Note that CNeRV results are comparable to NeRV, while autoencoders suffers from bluriness, due to the fact that without implicit representation it cannot perform well in the compression setting (tiny embedding).

A.3 Sensitivity Analysis and Limitations
lem, CNeRV still cannot reconstruct unseen images with perfect fidelity in this work. This also limits its application for visual codec or compression methods.

A.4 More visualization results
We show visualization results for seen images in Figure 11 and unseen images in Figure 12.
With the same embedding length and similar total size, CNeRV outperforms NeRV with better detail, absence of artifacts, and lack of spillover from previous frames. Although autoencoder with large embeddings can reach comparable generalization for unseen images, it struggles a lot for reconstruction of seen images. But, visual differences still exist between CNeRV and ground truth, and future work can focus on mitigating these issues. We provide more visualization results in a separate folder named 'cnerv_visualization', you can easily access them through the 'index.html'.
Ground Truth Autoencoder NeRV CNeRV Figure 12: Video compression results for unseen frames from Bunny. CNeRV results are similar for unseen as for seen, while autoencoders is still significantly blurrier than CNeRV. NeRV suffers from a myriad of failures.

A.5 NeRV Generalization Results
We provide more NeRV results on shuffled/sequential video frames in Table 14, with different number of training images on 'Big Buck Bunny'.

A.6 Embedding interpolation
We first provide interpolation results in pixel space and embedding space, together with ground truth embedding results, as shown in Table 15. Note that our interpolated embedding shows comparable PSNR with ground truth embedding on unseen images, which clearly demonstrates the internal generalization of our content-adaptive embedding.

A.8 Embedding Analysis
We also perform 3 measurements to compare the embeddings produced by CNeRV to prior work. We compute uniformity to examine the distribution of each set of embeddings on the hypersphere [? ].
With the same metric, we compute distances between neighbors for each method, to measure the extent to which each method encodes similar representations for neighboring frames. We also compute normalized distance, which is the neighbor distance (average distance between neighboring embeddings) divided by the uniformity (average distance between all embeddings). This normalized distance gives a more accurate reflection of the extent to which a given method's embeddings reflect a semantic connection between neighboring frames.
We compute linear centered kernel alignment (CKA) [? ] to compare the similarity between embeddings of different methods in a pairwise fashion, as introduced by [? ]. To compute this, we first obtain the matrices containing the embeddings for two different methods, such as NeRV and CNeRV, which we represent X and Y . We then compute the Gram matrices of the embedding matrices: K = XX T , L = YY T . The CKA value is given by the As Table 21 shows, while CNeRV's content adaptive embeddings are more tightly clustered than the embeddings for the other methods, CNeRV's neighboring embeddings are still relatively close to each other. CNeRV thus utilizes very little of the embedding space to generate meaningful and generalizable embeddings, and this is also consistent with embedding quantization results as in Figure 9. Figure 13 further reflects the difference between CNeRV embeddings and those of other methods. Whereas the convolutional autoencoders and the pixel-wise neural representations generate somewhat similar embeddings, CNeRV is quite unique by contrast. This shows how radically different our method is from those already in the literature, in spite of its comparable performance and desirable properties.       Figure 13: Embedding similarity for methods in Table 8. Embeddings of NeRV and CN-eRV are quite different from other methods.