{'fraction_non_alphanumeric': 0.022799066062354073, 'fraction_numerical': 0.002563750400586, 'mean_word_length': 5.1758552445575345, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Research on Image Super-resolution Reconstruction Based on Deep Learning

Image super-resolution processing technology refers to the technology of reconstructing high-resolution images by using low-resolution images with mutual displacement of multiple frames of the same scene. At present, image super-resolution processing technology is widely used in image reconstruction in various fields, such as: medical images, remote sensing images, robot vision, etc. In recent years, deep learning has also been widely used in various fields, especially in computer vision and speech processing. Since image super-resolution reconstruction the method of deep learning is introduced into the technology. Due to the good performance advantages of the deep learning method, it has become an important method for the research of image superresolution reconstruction technology. In this paper, the current status of domestic and foreign research on image super-resolution reconstruction is first simplified. Then, the theory of deep learning was explained, mainly the artificial neural network theory, the related theory of super-resolution reconstruction based on deep learning, and the method of image super-resolution reconstruction based on deep learning.


Introduction
Artificial neural network has super high learning ability, which can promote the rapid development of artificial intelligence technology, thus making it a hot spot in modern scientific research again [1]. With the continuous optimization of computer performance, the emergence of cloud computing technology and CPU, computing the amount has not been able to hinder the development of artificial neural networks [2]. In recent years, deep learning has made breakthrough progress in the field of image recognition, which has set off a wave of research in academia, especially in human-machine games. The current research on image super-resolution reconstruction has matured, but the use of deep learning methods to study image super-resolution reconstruction technology is relatively rare, so this should become the research direction of image super-resolution reconstruction technology in the next few years [3].

Research status of image super-resolution reconstruction at home and abroad
Image super-resolution reconstruction has undergone three development stages: reconstruction-based algorithms, example learning-based algorithms, and deep learning-based algorithms. Almost all reconstruction-based algorithms need to input multiple frames of the same scene while using the other two methods can break this limitation, you only need to input one frame. The following three phases are briefly introduced: (1) Reconstruction-based algorithm This algorithm mainly uses the techniques and methods of image processing and mathematical ideas to fuse and transform the obtained information of multiple frames of low-resolution images, so as to obtain a feature expression suitable for solving high-resolution images. Finally, then solve it. The commonly used algorithms are: iterative back projection method, convex set projection method and maximum posterior probability method.
(2) Example-based learning algorithm This algorithm combines low-resolution images with high-resolution images to form an image pair, then trains a dictionary of images, and then inputs the images to be reconstructed into the dictionary, compares and searches in the dictionary, and finds the corresponding high value of each sub-block highresolution image blocks, and finally get the reconstructed high-resolution image. The commonly used algorithms are: the method of finding the image by the manifold embedding technique proposed by Chang NH et.al, the anchor nearest neighbor regression method proposed by Timofte et.al, etc.
(3) Algorithms based on deep learning It is very difficult to reconstruct a single frame of image, and reasonable image feature expression is very important in the solution of this problem. The method of convolutional neural network is a more important method in current deep learning algorithms, which can convert low resolution the image is subjected to feature transformation and non-linear mapping process, so as to obtain more complex feature expressions of low-resolution images, and finally make the complex mapping relationship between low-resolution images and high-resolution images.
The deep network SRCNN, which was proposed in 2014, is the world's first image super-resolution reconstruction algorithm based on a deep convolutional network. It has only three convolutional layers, so the network structure is very simple, as shown in the figure below. With the rapid development of deep learning algorithms, image super-resolution algorithms based on convolutional neural networks have also developed rapidly, and various new network structures have gradually appeared.

Artificial neural networks
Artificial neural network is a mathematical model created by imitating the nerves of the brain. It is based on the principles of neural networks in biology, and has the characteristics of parallel distributed processing, high fault tolerance, and self-learning. Neurons are the most basic unit of a neural network is mainly composed of input weights, activation functions, and outputs. Neurons are also called perceptrons, it can not only perform simple Boolean operations, but also fit arbitrary linear functions and solve linear classification problems and regression problems.

Neural network basic theory
(1) The structural model of a simple neural network is shown in figure 2. The leftmost side is the input layer, the rightmost side is the output layer, and the middle part is the hidden layer. If it contains multiple hidden layers, the neural network is called a deep neural network. In the fitting of arbitrary functions, deep neural networks require more neurons in deep and shallow layers, while shallow neural networks require more neurons in shallow and wide layers. Generally speaking, deep neural networks save resources more but it is not easy to train, which requires a lot of data very good skills to train. (2) Learning methods The learning methods of neural networks can be divided into three categories: 1) Supervised learning: input the training sample data at the input of the neural network, find the difference between the actual output result and the expected answer to get the error signal, and then adjust the weight by controlling the error signal size, which can better adapt to the new environment.
2) Unsupervised learning: This method does not give standard data samples at the beginning, but puts the neural network directly into the environment and lets itself learn based on the characteristics of the given data.
3) Reinforcement learning: The input information of this method is between the first two methods. This does not need to give a specific label, but the output of the network should be judged so that the network parameters can be adjusted.
(3) Learning rules The learning rule refers to the process and method of modifying the neural network weights and bias values. It is mainly to train the neural network so that it can better fit our needs, so that special tasks can be performed completed. Frequently used are BP neural network algorithms, Hebb learning rules, etc.
(4) Activation function The activation function is mainly used to transform the data obtained by the neuron to reflect the characteristics of the neuron. The commonly used activation functions are: linear function, symbolic function, logarithmic function, hyperbolic tangent function, etc.

Related theories of super-resolution reconstruction based on deep learning
3.2.1. Basic theory of super-resolution reconstruction. The LR image in the image SR reconstruction is a single or multiple degraded images obtained by downsampling, blurring, and noise. SR reconstruction is based on the single or multiple LR images to restore the lost images during the image degradation process HR image. The blur and noise in the image degradation process correspond to the image deblurring and image denoising problems in the field of image restoration, and the focus in the SR reconstruction task is the spatial resolution, which is caused by the downsampling process. The decrease in resolution can also remove blurring and denoising. Obviously, the process of SR reconstruction is the inverse process of the process of image degradation. Therefore, understanding the degradation process of hr image to LR image is of great significance for analyzing SR reconstruction. In learning-based SR reconstruction, the prior knowledge of the hr image is very important. Understanding the degradation process and building an accurate mathematical model can provide corresponding prior knowledge. As shown in figure 3, the degradation of the HR image to the LR image flowchart.
In Equation (1), Y is the degraded image, X is the original image, S is the downsampling operator, B is the blurring factor, and v representing additive noise, the process of downsampling operator s and blurring factor b can be regarded as the process of image degradation as H. For SR reconstruction, finding the inverse process of the degradation process is an ideal goal, which is often difficult to achieve. Since deep learning was applied to SR reconstruction, a new idea has been proposed, that is, to train the deep network to find the mapping from LR image to HR image through the existing LR and HR image pairs. Since neural networks can theoretically fit arbitrary functions, therefore, in theory, an inverse mapping of an infinite number of LR and HR image pairs to obtain an ideal degraded model through training.

Convolutional neural network.
Convolutional neural networks are a type of neural network. They are developed from neural networks and are based on the same principles in forward propagation and back propagation of gradients. Therefore, understanding the principles of neural networks has certain implications for understanding convolutional neural networks significance.
Artificial neural network is the basis of deep learning. It simulates the structure of neurons in the human brain .The principle of forward propagation is similar to the transmission of neurotransmitters in neurons. Gradient back propagation has no biological basis but also shows its effectiveness. A single neuron is shown in figure 4. The input of the neuron is , the output is Y, the activation function is F, is the weight, B is the bias term, and the mathematical form of the neuron is expressed as: The algorithm built by the neural network is usually composed of a large number of neurons in layers, which is commonly known as a neural network model. A typical neural network model is shown in figure 5.
b h , b o the bias terms of the hidden layer and the output layer, respectively, H and O are the outputs of the hidden layer and the output layer, respectively. , the weight matrices of the input layer to the hidden layer and the hidden layer to the output layer, respectively, assuming that the resolution of the input image X is 64  64, then the number of weights is shared 64  64  h + h  q, it is obvious that the number of weights is too huge to train, so in the field of computer vision, convolutional neural networks are often used to process images. Back propagation is a method for neural networks to update weights, according to the chain rule, from the loss function along the structure of the neural network updates the weight from the output layer to the input layer. Take the neural network in figure5 as an example. Let the loss function be l, then the gradient is transmitted in The following way , where prod multiplies the input according to the situation to the shape of the matrix.
The convolution layer is the cornerstone of the breakthrough results of deep learning in the field of computer vision in recent years. It is a special network structure for image input based on neural networks. Its core part is a convolution kernel. Since each input of the neural network has a corresponding weight, the convolutional layer sets a convolution kernel of a fixed size and completes the input of the entire image through weight sharing. Therefore, the convolutional layer has two important characteristics: feeling receptive field means that when the image is input, the neuron is only connected to pixels within the range of the convolution kernel, so the output of the neuron is only affected by the elements in the range of the receptive field. Weight sharing means convolution the kernel slides across the entire picture by sliding, that is, the convolution kernel is shared by the entire picture, which greatly reduces the number of parameters. Convolution is a linear process, so convolution layers usually use non-linear activation such as ReLU, PReLU, the function enhances the non-linear mapping capability of the network model. The working principle of the convolutional layer is shown in figure 6.

Figure 6. Schematic diagram of feature extraction for convolutional layers
In figure 6, the lower matrix is the input and the upper matrix is the output, 33 a matrix of size Represents the convolution kernel. The convolution kernel slides through the entire input matrix according to the set step size to obtain the output matrix. The step size of figure6 is set to 2. The dashed line at the edge of the input matrix represents zero padding, which needs to be maintained during SR reconstruction. The size of the feature map does not change, and the corresponding zero-padding value is usually set according to the size and step size of the convolution kernel. The mathematical form of the convolution layer of a typical activation function is ReLU: Through the application of convolutional layers, deep learning networks can effectively extract features from a large number of data sets.

Advanced Deep Learning Network Structure
(1) Residual network With the development of deep learning, the number of layers of the network is increasing, and the deepening of the depth naturally makes the feature maps of convolutional layers in different locations can be divided into: low-level, intermediate-level, and high-level. Reviewing the imageNet over the years, large-scale visual recognition, the algorithm that won the championship in the ImageNet Large Scale Visual Recognition Competition (ILSVRC), the increase in accuracy in the classification task is Accompanied by the gradual deepening of the depth. Given the importance of the depth of the network, it is often inclined to design network models. However, the deepening of the network brings problems in training, such as the disappearance of gradients and the explosion of gradients, which hinders the convergence of the network and makes training more difficult. Another problem is that when the network reaches at a certain depth, it will face the problem of degradation: as the number of network layers increases, the accuracy reaches saturation, and adding more layers will even lead to performance degradation. HE proposed a residual network to solve the above problems, and the residual one of the advanced network design ideas in the network is hopping connections, as shown in figure 7: In theory, the mathematical form of residual learning is expressed as: In equation (5), X and Y are the inputs and outputs of the residual network (which can be part of the entire network). F (x,{Wi}) represents the learned residual mapping. As shown in figure 7, a residual network with two convolutional layers, , to activate the function, the bias Term is omitted for simplicity of representation. F (x) + x represents the summation operation of skip connections, and the summation of pixel-by-pixel sums, as the output of the residual network. Because skip connections do not introduce additional parameters and hardly introduce computational complexity, they can make the network easier to converge. It also solves the problem of degradation caused by increasing depth. Therefore, as an advanced network structure design method, the residual network, once proposed, wins the championship in the three tasks of classification, detection, and positioning of ilsvrc, and the number of layers in the network after reaching 152 layers, the convergence performance of the training is still guaranteed.
In SR reconstruction, since the degraded image and the real image are highly correlated in the image restoration task, the pixel values in most locations in the image have little or no difference in pixel values. The use of jump connections makes the network reconstructed by SR more sparse, and at the same time the input degraded image can be directly connected to the output directly and efficiently, which avoids the waste of bandwidth and calculation amount of the input degraded image information saved by the network.
(2) Densely connected network The residual network proves that shortening the distance between different layers through jump connections can make deep networks easier to train and get better results. However, as the network deepens, another problem is that the feature map of the convolutional layer contains information in the after many convolutional layers, it will face the problem of disappearance. In the SR reconstruction task, because the features such as details and textures in a small area of the image have an important influence Similarly, in the reverse direction during propagation, the gradient also easily disappears as it passes through many convolutional layers.
Huang proposed Densely Connected Convolutional Networks (DenseNets) to ensure the maximum information flow performance between different layers in the network and directly connect all layers in the network (with corresponding feature map dimensions) to each other. That is, each convolutional layer obtains additional input feature maps from all convolutional layers in the forward direction, and passes the output feature maps of that convolutional layer to all the convolutional layers in the backward direction as inputs. Sum with the residual network the operations are different, and the feature map is combined by channel in the dense connection.

Image super-resolution reconstruction based on deep learning
The main research directions of image super-resolution reconstruction based on deep learning are singleframe image super-resolution reconstruction and video super-resolution reconstruction. The following two directions will be introduced:

Single-frame image super-resolution reconstruction.
There are two main algorithms in this research direction: (1) Super-resolution reconstruction based on convolutional neural network (ARCNN) This algorithm is the first image super-resolution reconstruction algorithm using deep learning. It can directly learn the mapping relationship between low-resolution and high-resolution, thereby outputting high-resolution images. The traditional multi-module integrated super-resolution algorithm has more remarkable results. There are three main steps: 1) Extract image features, extract multiple image blocks from low-resolution images, and then use convolution operations to transform the image blocks into feature vectors. It is displayed and formed the feature matrix; 2) Non-linear mapping, the feature matrix obtained in the first step is mapped to the feature matrix of the high-resolution image; 3) The reconstructed image is combined with the previous step to complete the reconstruction of the highresolution image.
(2) Super-resolution reconstruction based on generative adversarial network (SRGAN) This algorithm is an algorithm constructed on the basis of generating an adversarial network. It can prove that although only the mean square error is used as a loss function to obtain a higher peak signalto-noise ratio, the reconstructed image is too smooth and high-frequency details are often lost. Therefore, the algorithm improves the loss function and uses the perceptual loss method to enhance the realism of the reconstructed image.

Video super-resolution reconstruction.
There are two main algorithms for video super-resolution reconstruction: (1) Video super-resolution reconstruction based on convolutional neural network (VSRnet) Consecutive frames are processed by motion compensation and then used as input to a convolutional Neural network, and then the convolutional layer is used to reconstruct the image. This algorithm proposes to shorten the training time by pre-training the model, and also uses continuous frame redundant information method, but the method of using motion compensation is not appropriate enough, and it cannot do real-time processing.
(2) Video super-resolution reconstruction combining motion compensation and spatiotemporal modules (VESPCN) This algorithm introduces motion compensation mechanism on the basis of sub-pixel convolutional network, so that real-time and multi-scale video super-resolution reconstruction can be realized. This algorithm is mainly composed of motion compensation module and spatio-temporal module. Realizing motion compensation can make redundant information of multiple frames input be effectively used, so that the effect of image reconstruction is better.

Conclusion
The research on image super-resolution reconstruction algorithms is very complicated, and it is timeconsuming and labor-intensive to calculate. With the rapid development of computer technology and the widespread application of deep learning methods, it is necessary to integrate it into the process of image super-resolution reconstruction. Reconstruction is faster and more stable. With the continuous advancement of technology, future super-resolution reconstruction methods based on deep learning will be applied to a wider range of fields to promote the development of society.