{'fraction_non_alphanumeric': 0.031423883578256526, 'fraction_numerical': 0.008203738051077186, 'mean_word_length': 5.187817258883249, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Single Image Internal Distribution Measurement Using Non-Local Variational Autoencoder

Deep learning-based super-resolution methods have shown great promise, especially for single image super-resolution (SISR) tasks. Despite the performance gain, these methods are limited due to their reliance on copious data for model training. In addition, supervised SISR solutions rely on local neighbourhood information focusing only on the feature learning processes for the reconstruction of low-dimensional images. Moreover, they fail to capitalize on global context due to their constrained receptive field. To combat these challenges, this paper proposes a novel image-specific solution, namely non-local variational autoencoder (\texttt{NLVAE}), to reconstruct a high-resolution (HR) image from a single low-resolution (LR) image without the need for any prior training. To harvest maximum details for various receptive regions and high-quality synthetic images, \texttt{NLVAE} is introduced as a self-supervised strategy that reconstructs high-resolution images using disentangled information from the non-local neighbourhood. Experimental results from seven benchmark datasets demonstrate the effectiveness of the \texttt{NLVAE} model. Moreover, our proposed model outperforms a number of baseline and state-of-the-art methods as confirmed through extensive qualitative and quantitative evaluations.


I. INTRODUCTION
Image super-resolution (SR) refers to the task of recovering a latent high-resolution (HR) image from a corresponding lowresolution (LR) image. This has been one of the most widely explored inverse problems in computer vision [1], [2]. A LR image I x is assumed to be modeled as the output of the following degradation: where D defines a degradation mapping function, I y is the corresponding HR image, and λ denotes the degradation parameter. The higher the degradation, the harder becomes  the task to reconstruct the HR image [3]. The image superresolution problem is also explored in the fields of remote sensing [4], surveillance imaging [5], and medical imaging [6]. While a number of approaches have been attempted to solve this problem, it remains ill-posed particularly since any specific LR image may correspond to croppings from multiple HR counterparts. Three types of solutions are usually provided to solve the SR problem; interpolation-based [7], learningbased [8], [9], [10] and reconstruction-based [11], [12], [13].
Learning-based SR methods learn the non-linear mapping between HR and LR image using probabilistic generative models, random forest, linear or non-linear models, neighbor embedding [14] and sparse regression [15]. Interpolationbased methods utilize the adjacent pixels to calculate the interpolated pixels by using an interpolation kernel. Several types of existing interpolation-based models have been used to tackle the SR problem such as bicubic [16], edge-directed estimation [17], and auto-regressive models [18]. Interpolationbased methods are very computationally efficient and relatively simple than other architectures. However, these methods suffer from low accuracy compared to other methods because of poor representation learning capacity [19].
Learning-based solutions are widely used in the SR task. They can primarily be categorized into three types: Codebased [20], [21], [9], CNN-based [22], [23] and regressionbased [24]. In the past, learning-based solutions have shown great success for image super-resolution due to a robust feature learning capability [25], [26]. The regression-based solutions are much faster than any other methods but, compared to other learning-based methods, produce blurry images and low peak-signal-noise ratio, due to poor representation learning [15], [10]. Learning-based solutions generally measure the similarity between LR & HR images. A number of methods have been proposed to solve this problem but among them, CNN-based methods have superior performance because of their robust representation learning capabilities [27]. As a learning-based method, a hierarchical pyramid structure was developed using residual layers for image super-resolution [28].
Zhang et al. [29] introduced residual dense block in order to learn hierarchical feature maps, utilizing a bottleneck layer at the end of the residual dense layer. The local connection in the same block represents short-term memory and skip connections define long-term memory for representation learning. Lim et al. [30] proposed Enhanced Deep Super-Resolution (EDSR) with a huge improvement in performance utilizing the residual structure. Despite the excellent performance, these methods utilize residual structure that is computationally resourceful. In [31], a combination of channel-wise attention and spatial attention block was developed for single image super-resolution (SISR). Both of these blocks were again combined with residual creating robust SR methods. This method captures sufficient representations from feature space and suppresses irrelevant information. As this approach combines two kinds of attention block with a stacked neural network, it consumes large amounts of memory and slows down the training process.
In [32], an autoencoder-based SISR method has been introduced with symmetric skip connections. Similarly, Tai et al. [33] featured a deep recursive residual network (DRRN) utilizing memory block with residual representation learning. These solutions provide better results with the help of large scale architecture. Park et al. [34] introduced a high dynamic range for super-resolution task that is very lightweight and easy to implement but decomposing the image causes loss of important information which results in low peak-signal noise ratio (PSNR) values.
Many upsampling strategies have been observed in the literature. Efficient sub-pixel CNN (ESPCN) uses sub-pixel convolution for upsampling [35] that stores channel information for extra points then reorganizes those points for HR reconstruction. Fast super-resolution CNN (FSRCNN) utilizes deconvolution operation to upsample [36]. Hua et al. used a deconvolution operation for the upsampling process, featuring the arbitrary interpolation operator and subsequent convolution operator [37]. It is to be noted that the deconvolution operation has two disadvantages: deconvolution is used at the end of the network, and the downsampling kernel is not known. Unknown input estimation consequently results in poor performance. To avoid these issues, we utilize linear upsampling of LR images so that we only focus on reconstruction quality rather than upsampling kernels.
From a theoretical perspective, it can be deduced that the deeper neural architecture provides better result than shallow architecture [38]. Keeping this in mind, Kim et al. [22] first proposed a very deep architecture for SISR task. With 20 layers, VGGNet uses 3x kernels for all layers. Additionally, this method uses a high learning rate for faster convergence and utilizes gradient clipping to alleviate the gradient explosion problem. To learning short-term memory information, skip connections have been used in many tasks. Another work introduced recursive topology with parameter reduction using recursive convolution kernel [33]. However, these settings are risky as self-supervised settings because a faster learning rate will provide shallow feature learning process, thus resulting in poor performance.
Several reconstruction-based SISR methods have been introduced to solve the SISR problem, utilizing a shallow feature learning process [39], [40]. KernelGAN [41], consisting of a deep linear generator and a discriminator, supports blind SISR. A deep linear generator removes non-linear activation functions, but the overall loss function is not convex. The discriminator uses fully convolutional layers with no strides and pooling. Even though the overall structure means that the model converges faster, it is still difficult to obtain the global minimum. Our method utilizes the non-linear activation function in both encoder and decoder, making network learn more intuitive information than KernelGAN. Shaham et al. [42] proposed a multidisciplinary generative model capable of performing multiple computer vision tasks. To our knowledge, this work was the first attempt to use an unconditional generative model for the ZSSR task. It utilizes an adversarial network as a reconstruction-based method learning only abstract features from image patches. Due to the complexity of training an adversarial network, GAN models often suffer from convergence failure and mode collapse. Moreover, training adversarial training takes longer than discriminative models.
To alleviate these problems, we have devised an imagespecific architecture, called probabilistic non-local variational neural autoencoder (NLVAE), which can generate high-quality images with a robust pixel learning capability. Our generative solution is specifically designed for ZSSR, storing more disentangled and intuitive features and learning from lowdimensional space.
Our specific contributions can be summarized as follows: • An unconventional internal method has been introduced for the ZSSR task where only one one LR image and its corresponding HR image are required for the training process. The proposed method is comnpletely unsupervised and does not require any prior training. It establishes a new state-of-the-art (SOTA) which outperforms currently available methods. • The proposed light-weight non-local feature extraction module harvests maximum representations from different receptive regions boosting the super-resolution performance. • The proposed loss function aids to reconstruct high quality images by controlling the Lagrange multiplier and marginal value. The rest of the paper is organized as follows. Section II shows some works related to our proposed network structure. Section III describes the working principle of our method.
In section IV, we provide quantitative and qualitative results using our model. Section V provides some ablation studies demonstrating the robustness of our network and section VI discusses the limitation of our strategy along with similarities and dissimilarities with other methods. Section VII provides concluding remarks.

II. RELATED WORK
Generative Models. Generative models have been proven to reconstruct finer texture details and are able to generate more photo-realistic images than CNN-based methods. While shallow CNN-based SR methods provide detailed low-frequency information, GAN-based methods as generative models can discover high-frequency information. Super-Resolution GAN (SRGAN) [43] makes use of perceptual loss as well as residual dense network generating high-resolution images. Wang et al. [44] proposed residual-in-residual without batch normalization which produced HR images through adversarial training. Majdabad et al. [45] attached a capsule network as a complex network with GAN for face super-resolution. In [46], a conditional GAN has been introduced using groundtruth as a conditional variable for the discriminator. Similar to conditional GAN, conditional autoregressive generative models utilize maximum likelihood estimation depending on conditions. Based on these conditions, the generated HR images are reconstructed based on previous learned pixels [47], [48]. However, these generative models suffer from mode collapse and convergence failure problems [49]. Moreover, these methods are computationally expensive and integration with self-supervised training is quite difficult to implement as GAN-based methods require more image data for training than learning-based methods [50], [44].
Non-Local Networks. Non-local networks usually comprise an attention module with non-local blocks. Wang et al. [51] proposed a deformable non-local attention module for video super-resolution. In [52], a non-local recurrent model was introduced for SISR task, which can learn deep feature correlation among neighbourhood locations of patches. Zhang et al. [29] featured residual network with non-local attention units for image super-resolution. Another work presents a cross-scale non-local attention module for learning intrinsic feature correlations of images [53].
Zero-Shot Super-Resolution Methods. Shocher et al. [54] introduced the term "ZSSR," presenting a shallow CNN model to learn the probability distribution of the LR and HR images. The major disadvantage of this network is that it extracts local features utilizing a simple CNN architecture and a shallow CNN model, which also results in poor performance. Another internal method proposed in [55], introducing Deep Image Prior (DIP) to build a bridge between a CNN and convolutional sparse coding. The solution takes the neural network as the output of the reconstruction and random input signals. This is the first approach which creates a bridge between a code-based method and a learning-based method for ZSSR. Untrained DIP basically focuses on the smaller receptive fields for intuitive neural representation, but loses more context as feature extraction is limited to the smaller regions. Fig. 1 depicts that DIP Concatenation Convolution + Batch Norm + LeakyReLU The non-local block is composed of 3× kernels and 1× kernels. The initial convolution kernel is concatenated with the last feature transform to learn relative positional features. S×S defines the spatial size of the feature and the channel information is denoted as C method shows very weak structural information compared to NLVAE. Due to a weak feature extraction process, this method suffer from low accuracy in terms of performance metrics [56].

III. NETWORK STRUCTURE
In this section, we demonstrate the structure of our proposed non-local block in the neural encoder and decoder. We also show the measurement of posterior distribution and loss function and we provide a full analysis of how the Lagrange multiplier controls the reconstruction quality of generated image.

A. Non-Local Encoder-decoder
As shown in Fig. 3, our proposed NLVAE model consists of an encoder and a decoder consisting of non-local convolution blocks. Fig. 2 depicts the overview of non-local blocks. The non-local block utilized in NLVAE can exploit spatial correlation between neighbourhood and locations. To design a computationally effective spectral correlation module, we have overlooked residual structure. Each non-local block is composed of convolution blocks and each convolution block comprises of 2D convolution, point-wise convolution and batch normalization [57] followed by Leaky-ReLU activation function. The encoder encodes the input image x into the latent representation z ( z = ψ(x)) and the decoder reconstructs the representation back to its approximate original data. We assume that the low-resolution image is an input vector denoted by x and z denotes the latent representation. The latent variables are controlled by a Gaussian distribution along with a diagonal covariance matrix. The latent space dimension is denoted by J. The output of the non-local convolutional encoder comprises a mean µ and a log of variances log(σ). Through the reparameterization trick, a noise vector (∈) is obtained from the latent space [58]. The goal of the NLVAE model is the ability to produce a high-resolution reconstructed image from the low-resolution image, exploiting the relationship between the input vector and prior distribution p(z). Algorithm 1: Training NLVAE model Input : Initialize network parameters while not converged do X = psuedo labels of single L r image Z p ← Distribution from prior N(0, I)

B. Posterior Distribution
We denote the low-resolution input data distribution as x p d x and the high-resolution reconstructed data distribution asx p(x). The encoded and decoded data distributions are represented as q φ (z|x) and p θ (z|x) respectively, where φ and θ are the variables of the encoder and decoder networks. The q(x) tries to approximate the output prior p(z). Centered isotropic multivariate Gaussian N (0, I) was chosen as the prior p(z) over the latent variables [59]. The inference model is designed to output two individual variables µ and σ, and thus the posterior q φ (z|x) = N (z; µ, σ 2 ). With this setting, to get the desired prior distribution, the non-local convolutional encoder and decoder are trained to optimize the reconstruction error (that is, the mean squared error). The loss function tries to approximate between each patch of LR and HR over a fake minibatch x n where n is the size of the fake minibatch. The total number of data points is denoted as N .

C. Loss Function
It is useful to prepare the low-resolution input image by clustering in latent space, eradicating the noise L R loss is summed over the data points and the average of the fake minibatch is calculated. Thus, it provides more weight to the reconstruction error helping reduce potential model collapse.
To obtain the desired prior distribution, KL divergence is utilized on the encoded variable to measure the probability distance of LR and HR images. KL divergence is calculated over the fake minibatch as (4) Therefore, the total loss is calculated as where β denotes the Lagrangian multiplier and α denotes the marginal value. As the negation of L BET A is the lower bound of the Lagrangian, minimization of the loss is equivalent to maximization of the Lagrangian, which is useful for our initial optimization problem. The α controls the quality of image reconstruction as an aid to the objective function. For β = 1, the working principle is the same as traditional VAE. When β > 1, it applies a stronger constraint on the latent bottleneck and limits the representation capacity of z [60]. Maintaining the disentanglement is the most effective representation for some of the conditionally independent generative factors [61].

D. Lagrangian multiplier variation for different upscaling
The addition of β in VAE provides more disentangled information and sharp gradients compared to traditional VAE [62].
A higher value of β provides more efficient encoded latent vectors and further encourages disentanglement. However, too large β may lead to poorer reconstruction quality as it creates a trade-off with the extent of disentanglement. The reconstruction loss ensures the network captures useful information while forming the latent distribution. An increase in the number of latent variables reduces image quality, thus through empirical evaluation, we selected a different Lagrangian multiplier for different upscaling. For our experimental settings, we have selected 150, 200, and 300 for 3 ×, 4 ×, and 8 × upscaling factor respectively.

E. Computational Efficiency of Non-local Block
In this subsection, the computational efficiency of the nonlocal block is briefly explained. The point-wise convolution is the core of non-local block for calibrating spatial information. It also serves as the channel reduction technique in this network. The weights of the point-wise convolution can be calculated as: For this operation, K = 1. Then Equation 6 becomes: And the corresponding number of operations is therefore: For the standard convolution operation, the number of weights will be: And the corresponding number of operation is: Now, the reduction factors of weights and operations can be defined as: From the reduction factors of weights and operation, we can observe the reduction in computational cost due to the use of point-wise convolutions. For the standard convolution operation, the number of weights will be And the corresponding number of operation is, Now, the reduction factors of weights and operations can be defined as From the reduction factors of weights and operation, we can observe the reduction in computational cost due to the use of point-wise convolutions.

B. Implementation Details & Training Settings
We make use of the TensorFlow framework with Python for all the experiments. The experiments are implemented on a a Nvidia GeForce GTX Titan X and Intel Xeon CPU at 2.40 GHz machine. All the images are resized to 256 × 256. The Adam optimizer [72] is utilized with β 1 = 0.9, β 2 = 0.999, and σ = 10 −8 . Pesudo labels are created for training purposes as there exists only one single image. The model is trained till 2000 epochs. We use the L2 loss function for our solution. For our settings, the hyperparameters are selected empirically. We perform the experiments for three different scaling factors-3×, 4×, and 8×. The value of β is set to 500 for all the experiments.
As the proposed method utilizes self-training strategy, it takes both L R and H R single image as input for training pipeline. Then, it tries to learn the relationship leveraging nonlocal attention blocks. Finally, the self-training model tries to generate a single H R image from the pre-trained weights. It is to be noted that the performance metric calculation is the mean of all single generated images for each datasets. Table I ESRGAN) have been compared against the proposed framework. It is worth mentioning that deeper architectures perform better over the shallower networks. We also note that larger scaling factors affect the performance of the existing external methods. Among learning-based methods, MemNet demonstrates good performance over large scaling factors because of its large architecture, but its performance drops when the scaling factor is relatively small. The reconstruction-based strategies have higher structural similarity than other methods. Most importantly, our proposed NLVAE model outperforms    other reconstruction-based methods in all the scaling factors, generating high-resolution photo-realistic images. This justifies the incorporation of the non-local convolutional block which enables the model to perform better, specifically, on smaller scaled images. Moreover, the deeper architecture of the generative models enhances the performance on large scaling factors, leading the network to a robust zero-shot superresolution network.

1) Quantitative:
2) Qualitative: Fig. 5 and Fig. 4 depict the visualization of learning-based methods and reconstruction-based methods respectively. Samples from the Set14 and General100 datasets have been used in order to visualize the learning-based solutions. It is confirmed from Fig. 5 that our solution produces sharp edges and avoids any undesirable artifacts. As can be seen, a featured generative solution learns better representations between an LR image & its corresponding HR image. For a fair comparison with reconstruction-based solutions, we utilized Set5, Urban100, and BSDS100 for qualitative comparison among generative SISR models. The visual property of the reconstructed images is magnificent compared to other methods because of its global contextual feature learning process. Fig. 4 demonstrates that our method can reduce the blurring artifacts presenting a powerful feature learning ability. It is imperative that the NLVAE model provides better details in regions of irregular structures. More detailed visualizations containing random HR samples from the generated sets and real datasets are provided in the supplementary material.

V. ABLATION STUDY A. Loss function & Optimizers
In Fig. 6(a), we have explored different loss functions and optimizers to evaluate the performance of our proposed model. We observe that the combination L2 + Adam converges to gradient more smoothly than any other solutions. Among all optimizers, Adam converges to gradient faster than others. SGD and RMSProp provide competitive results but developing slower solutions for a zero-shot process. Between L2 and L1 loss functions, L2 provides faster training and finer reconstruction quality. All the hyperparamters are fixed for this ablation study.

B. Feature Extraction Blocks
To verify the robustness of our non-local convolutional block, we explored various feature extraction units. Fig. 6(b) shows that non-local convolutional unit performs better than other feature learning units. We observe that the residual unit learns slightly better representations than other units but has relatively larger computational burdens. Comparing our non-local unit against other traditional convolution operations (including depth-wise separable convolution, transposed convolution, and standard convolution operations) our method shows excellent performance with the lowest MSE between LR & HR images.

C. Feature Extraction Blocks
To verify the robustness of our non-local convolutional block, we explored various feature extraction units. Fig. 6(b) shows that non-local convolutional unit performs better than other feature learning units. We observe that the residual unit learns slightly better representations than other units but has relatively larger computational burdens. Comparing our non-local unit against other traditional convolution operations (including depth-wise separable convolution, transposed convolution, and standard convolution operations) our method shows excellent performance with the lowest MSE between LR & HR images.

D. Non-Local Blocks
In this ablations, we study the essentially of non-local blocks for image generation. In table II, the PSNR and SSIM values are depicted against number of non-local blocks for our proposed method. We note that an increase in non-local blocks provide more accurate image but also increases computational resources. Moreover, we note unusual instability using 5 or more non-local blocks with ADAM optimizer.

VI. DISCUSSIONS
In this subsection, we discuss the similarity, dissimilarities and limitations of our method compared to other data-driven strategies. Table III shows that the input image is linearly upscaled before processing for super-resolution. Similar to VDSR and DRCN, we also upscale the lowresolution image but linearly. The reconstruction process in our method is progressive as we combine both learning-based and reconstruction-based methods. Learning-based methods generally utilized direct reconstruction of HR images. We adopt the L2 loss function for faster convergence in order to maintain high-reconstruction quality. As mentioned above, we do not use a residual representation learning process due to the computational cost for self-supervised settings. Our settings use small modifications of self-supervised settings. We do not use batches of images per epochs; instead we utilize fake batches of a single image for every epochs. Moreover, we perform all these experiments on different sizes of dataset to explore structural variation. Experiments have done on small datasets (Set5, Set14) as well as large datasets (Manga109, Urban100, BSD100) to justify the performance of our proposed solution. VII. CONCLUSIONS We have presented NLVAE, an untrained generative model, featuring a neural encoder-decoder framework capable of reconstructing high-resolution images. With the use of nonlocal convolutional modules, the model is enabled to capture high-quality semantic information. In addition, the beta variational autoencoder provides more disentangled information reconstructing high-resolution images. Combining the learningbased and reconstruction-based methods, the present method generates sharp and photo-realistic images. The effectiveness of the present model has been confirmed through an extensive experimentation compared with a number of SOTA methods, both qualitatively and quantitatively on multiple benchmark datasets. Moreover, leveraging the power of robust feature learning and generative modeling, the proposed model obviates the need for a large scale dataset while performing SISR. It is to be noted that our proposed method relies on linear upsampling before the super-resolution task. Our future work will include further validation of the NLVAE model against more challenging data settings across various domains as well as more powerful automatic upsampling strategy. We envision more extensive comprehension of our model and more intuitive design of the objective function .