{'fraction_non_alphanumeric': 0.04007884362680683, 'fraction_numerical': 0.01222806249087458, 'mean_word_length': 4.701768990634755, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Texture Hallucination for Large-Scale Painting Super-Resolution

We aim to super-resolve digital paintings, synthesizing realistic details from high-resolution reference painting materials for very large scaling factors (e.g., 8x, 16x). However, previous single image super-resolution (SISR) methods would either lose textural details or introduce unpleasing artifacts. On the other hand, reference-based SR (Ref-SR) methods can transfer textures to some extent, but is still impractical to handle very large scales and keep fidelity with original input. To solve these problems, we propose an efficient high-resolution hallucination network for very large scaling factors with efficient network structure and feature transferring. To transfer more detailed textures, we design a wavelet texture loss, which helps to enhance more high-frequency components. At the same time, to reduce the smoothing effect brought by the image reconstruction loss, we further relax the reconstruction constraint with a degradation loss which ensures the consistency between downscaled super-resolution results and low-resolution inputs. We also collected a high-resolution (e.g., 4K resolution) painting dataset PaintHD by considering both physical size and image resolution. We demonstrate the effectiveness of our method with extensive experiments on PaintHD by comparing with SISR and Ref-SR state-of-the-art methods.


Introduction
Super-resolution for digital painting images has important values in both culture and research aspects. Many historical masterpieces were damaged and their digital replications are in low-res, low-quality due to technological limitation in old days. Recovery of their fine details is crucial for maintaining and protecting human heritage. It is also a valuable research problem for computer scientists to restore high-res painting due to the rich content and texture of paintings in varying scales. Solving this problem could lay the cornerstone for the more challenging recovery of general natural images.
Super-resolving painting images is particularly challenging as vary large upscaling factors (8×, 16×, or even larger) are required to recover the brush and canvas details of artworks, so that a viewer can fully appreciate the aesthetics as from the original painting. Fortunately, there is a big abundance of existing artworks scanned in high-resolution, which provide the references for the common texture details shared among most paintings. Therefore, in this paper we formulate a reference-based super-resolution problem setting for digital paintings with large upscaling factors.
To this end, we collect a large-scale high-quality dataset for oil painting with diverse contents and styles. We explore new deep network architectures with efficient memory usage and run time for large upscaling factors, and design wavelet-based texture loss and degradation loss to achieve high perceptual quality and fidelity at the same time. Our proposed method can hallucinate realistic details based on the given reference images, which is especially desired for large factor image upscaling. Compared to the previous state-of-the-arts, our proposed method achieves significantly improved visual results, which is verified in our human subjective evaluation. Our trained model can also be applied on natural photos and generates promising results. In Fig. 1, we compare with other state-of-the-art methods for large scaling factors (e.g., 16×).
In summary, the main contributions of this work are: • We proposed a reference-based image super-resolution framework for large upscaling factors (e.g., 8× and 16×) with novel training objectives. Specifically, we proposed wavelet texture loss and degradation loss, which allow to transfer more detailed textures.
• We collected a new digital painting dataset PaintHD with high-quality images and detailed meta information, by considering both physical and resolution sizes.

Related Work
Recent work on deep-learning-based methods for image super-resolution [22,24,20,35,36,37] is clearly outperforming more traditional methods [9,3,27] in terms of either PSNR/SSIM or visual quality. In this Section we will focus on the former for conciseness.

Single Image Super-Resolution
Single image super-resolution (SISR) recovers a highresolution image directly from its low-resolution (LR) counterpart. The pioneering SRCNN proposed by Dong et al. [5], made the breakthrough of introducing deep learning to SISR, achieving superior performance than traditional methods. Inspired by this seminal work, many representative works [29,16,17,6,25,22,35] were proposed to further explore the potential of deep learning and have continuously raised the baseline performance of SISR. In SR-CNN and followups VDSR [16] and DRCN [17], the in-put LR image is upscaled to the target size through interpolation before fed into the network for recovery of details. Later works demonstrated that extracting features from LR directly and learning the upscaling process would improve both quality and efficiency. For example, Dong et al. [6] provide the LR image directly to the network and use a deconvolution for upscaling. Shi et al. [25] further speed up the upscaling process using sub-pixel convolutions, which became widely adopted in recent works. Current state-of-the-art performance is achieved by EDSR [22] and RCAN [35]. EDSR takes inspiration from ResNet [13], using long-skip and sub-pix convolutions to achieve stronger edge and finer texture. RCAN introduced channel attention to enforce better learning of high-frequency information.
Once larger upscaling factors were achievable, e.g., 4×, 8×, many empirical studies [20,24,37] demonstrated that the commonly used quality measurements PSNR and SSIM proved to be not representative of visual quality, i.e., higher visual quality may result in lower PSNR; a fact first investigated by Johnson et al. [15] and Ledig et al. [20]. The former investigated perceptual loss using VGG [26], while the later proposed SRGAN by introducing GAN [11] loss into SISR, which boosted significantly the visual quality compared to previous works. Based on SRGAN [20], Sajjadi et al. [24] further adopted texture loss to enhance textural reality. Along with higher visual quality, those GAN-based SR methods also introduce artifacts or new textures synthesized depending on the content, which would contribute to increased perceived fidelity at the expense of reduced PSNR.
Although SISR has been studied for decades, it is still limited by its ill-posed nature, making it difficult to recover fine texture detail for upscaling factors of 8× or 16×. So, most existing SISR methods are limited to a maximum of 4×. Otherwise, they suffer serious degradation of quality. Works that attempted to achieve 8× upscaling, e.g., Lap-SRN [19] and RCAN [35], found visual quality would degrade quadratically with the increase of upscaling factor.

Reference-based Super-Resolution
Different from SISR, reference-based SR (Ref-SR) methods attempt to utilize self or external information to enhance the texture. Freeman et al. [8] proposed the first work on Ref-SR, which replaced LR patches with fitting HR ones from a database/dictionary. [7,14] considered the input LR image itself as the database, from which references were extracted to enhance textures. These methods benefit the most from repeated patterns with perspective transformation. Light field imaging is an area of interest for Ref-SR, where HR references can be captured along the LR light field, just with a small offset. Thus, making easier to align the reference to the LR input, facilitating the transfer of high-frequency information in [2,38]. CrossNet [39] took advantage of deep learning to align the input and ref-erence by estimating the flow between them and achieved state-of-the-art performance.
A more generic scenario for Ref-SR is to relax the constraints on references, i.e., the references could present large spacial/color shift from the input. More extremely, references and inputs could contain unrelated content. Sun et al. [27] used global scene descriptors and internet-scale image databases to find similar scenes that provide ideal example textures. Yue et al. [32] proposed a similar idea, retrieving similar images from the web and performing global registration and local matching. Recent works [30,37] leveraged deep models and significantly improved Ref-SR performance, e.g., visual quality and generalization capacity.
Our proposed method further extends the feasible scaling factor of previous Ref-SR methods from 4× to 16×. More importantly, as oppose to the previous approach [37] which transfers the high-frequency information from reference as a style transfer task, we conduct texture transfer only in high-frequency band, which reduces the transfer effect (and hence the distortion) on the low-frequency content.

Approach
We aim to hallucinate the super-resolution (SR) image I SR for large scaling factor s from its low-resolution (LR) input I LR and transfer highly detailed textures from highresolution (HR) reference I Ref . However, most previous Ref-SR methods [39,37] could mainly handle relatively small scaling factors (e.g., ×4). To achieve visually pleasing I SR with larger scaling factors, we firstly build a more compact network (see Fig. 2) and then apply novel loss functions to the output.

Pipeline
We first define L levels according to scaling factor s, where s = 2 L . Inspired by SRNTT [37], we conduct texture swapping in the feature space to transfer highly detailed textures to the output (Fig. 2). The feature upscaler acts as the mainstream of upscaling the input LR image. Meanwhile, the reference feature that carries richer texture is extracted by the deep feature extractor. At the finest layer (largest scale) the reference feature is transferred to the output.
As demonstrated in recent works [21,35], the batch normalization (BN) layers commonly used in deep models for stabilizing the training process turns to degrade the SR performance. Therefore, we avoid BN layer in our feature upscaling model. More importantly, the GPU memory usage is largely reduced, as the BN layer consumes similar amount of GPU memory as convolutional layer [22].
To efficiently transfer high-frequency information from the reference, we swap features at the finest level L, where the reference features are swapped according to the local feature matching between the input and reference. Since patch matching is time-consuming, it is conducted at lower is a neural feature extractor (e.g., VGG19 [26]) matching the same level. I LR↑ is upscaled by Bicubic interpolation with scaling factor s. To match the frequency band of I LR↑ , we first downscale and then upscale it with scaling factor s. For each patch from φ L−2 (I LR↑ ), we could find its best matched patch from φ L−2 (I Ref ↓↑ ) with highest similarity.
Then, using the matching information M L−2 , we transfer features at level L and obtain the new feature F L T via where H L T (·) denotes feature transfer operation. φ L (I Ref ) extracts neural feature from the high-resolution reference On the other hand, we also extract deep feature from the LR input I LR and upscale it with scaling factor s. Let's denote the upscaled input feature as F SR and the operation as H F SR (·), namely F SR = H F SR (I LR ). To introduce the transferred feature F L T into the image hallucination, we fuse F L T and F SR by using residual learning, and finally reconstruct the output I SR . Such a process can be expressed as follows where [F SR , F L T ] refers to channel-wise concatenation, H Res (·) denotes several residual blocks, and H Rec (·) denotes a reconstruction layer.
Although we can already achieve super-resolved results with larger scaling factors by using the above simplifications and improvements, we still aim to transfer highlydetailed texture from reference even in such challenging cases (i.e., very large scaling factors). For that, we propose wavelet texture and degradation losses.

Wavelet Texture Loss
Motivation. Textures are mainly composed of highfrequency components. LR images contain less highfrequency components, when the scaling factor goes larger. If we apply the loss functions (including texture loss) on the color image space, it's still hard to recover or transfer more high-frequency ones. However, if we pay more attention to the high-frequency components and relax the reconstruction of color image space, such an issue could be alleviated better. More specifically, we aim to transfer as many textures as possible from reference by applying texture loss on the high-frequency components. Wavelet is a proper way to decompose the signal into different bands with different levels of frequency components.
Haar wavelet. Inspired by the excellent WCT 2 [31], where a wavelet-corrected transfer was proposed, we firstly apply Haar wavelet to obtain different components. The Haar wavelet transformation has four kernels, LL T , LH T , HL T , HH T , where L T and H T denote the low and high pass filters, As a result, such a wavelet operation would split the signal into four channels, capturing low-frequency and highfrequency components. We denote the extraction operations for these four channels as H LL W (·), H LH W (·), H HL W (·), and H HH W (·) respectively. Then, we aim to pay more attention to the recovery of high-frequency components with the usage of wavelet texture loss.
Wavelet texture loss. As investigated in WCT 2 [31], in Haar wavelet, the low-pass filter can extract smooth surface and parts of texture and high-pass filters capture higher frequency components (e.g., horizontal, vertical, and diagonal edge like textures).
Ideally, it's a wise choice to apply texture loss on each channel split by Haar wavelet. However, as we calculate texture loss in different scales, such a choice would suffer from very heavy GPU memory usage and running time. Moreover, as it's very difficult for the network to transfer highly detailed texture with very large scaling factors, focusing on the reconstruction of more desired parts would be a better choice. Consequently, we propose a wavelet texture loss with HH kernel and formulate it as follows where H HH W (·) extracts high-frequency component from the upscaled output I SR with HH kernel. F l T is the transferred feature in feature map space of φ l . Gr(·) calculates Up Down ↓ Deg. loss Figure 3: The illustration of our proposed degradation loss. We try to minimize the degradation loss L deg between the downscaled output I SR↓ and the original input I LR .
the Gram matrix for each level l, where λ l is the corresponding normalization weight. · F denotes Frobenius norm.
As shown in Eq. (5), we mainly focus on the texture reconstruction of higher frequency components, which would transfer more textures with somehow creative ability. Then, we further relax the reconstruction constraint by proposing a degradation loss.

Degradation Loss
Motivation. Most previous single image SR methods (e.g., RCAN [35]) mainly concentrate on minimizing the loss between the upscaled image I SR and ground truth I GT . For small scaling factors (e.g., ×2), those methods would achieve excellent results with very high PSNR values. However, when the scaling factor goes very large (e.g., 16×), the results of those methods would suffer from heavy smoothing artifacts and lack favorable textures (see Fig. 1). On the other hand, as we try to transfer textures to the results as many as possible, emphasizing on the overall reconstruction in the upscaled image may also smooth some transferred textures. To alleviate such texture oversmoothing artifacts, we turn to additionally introduce the LR input I LR into network optimization.
Degradation loss. Different from image SR, which is more challenging to obtain favorable results, image downscaling could be relatively easier. It's possible to learn a degradation network H D , that maps the HR image to an LR one. We train such a network by using HR ground truth I GT as input and try to minimize the loss between its output H D (I GT ) and the original LR counterpart I LR .
With the degradation network H D , we are able to mimic the degradation process from I GT to I LR , which can be a many-to-one case. Namely, there exists many upscaled images corresponding to the original LR image I LR , which helps to relax the constraints on the reconstruction. To make use of this property, we try to narrow the gap between the downscaled output I SR↓ and the original LR input I LR . As shown in Fig. 3, we formulate it as a degradation loss where I SR↓ denotes the downscaled image from I SR with scaling factor s and · 1 denotes 1 -norm. With the pro-  [35], SRNTT [37], and our method on CUFED5. Our result is visually more pleasing than others, and generates plausible texture details.
posed loss functions, we further give details about the implementation.
where φ 5,1 extracts N 5,1 feature maps from 1-st convolutional layer before 5-th max-pooling layer of the VGG-19 [26] network. φ 5,1 i is the i-th feature map. We also adopt WGAN-GP [12] for adversarial training [11], which can be expressed as follows where G and D denote generator and discriminator respectively, and I SR = G(I LR ). P r and P g represent data and model distributions. For simplicity, here, we mainly focus on the adversarial loss for generator and show it as follows Training. The weights for L rec , L tex , L deg , L per , and L adv are 1, 10 −4 , 1, 10 −4 , and 10 −6 respectively. To stabilize the training process, we pre-train the network for 2 epochs with L rec and L tex . Then, all the losses are applied to train another 20 epochs. We implement our model with TensorFlow and apply Adam optimizer [18] with learning rate 10 −4 .

Dataset
For large upscaling factors, e.g., 8× and 16×, input images with small size, e.g., 30×30, but with rich texture in its originally HR counterpart will significantly increase the arbitrariness/smoothness for texture recovery because fewer pixels result in looser content constraints on the texture recovery. Existing datasets for Ref-SR are unsuitable for such large upscaling factors (see Fig. 4). Therefore, we collect a new dataset of high-resolution painting images that carry rich and diverse stroke and canvas texture.
The new dataset, named PaintHD, is sourced from the Google Art Project [4], which is a collection of very large zoom-able images. In total, we collected over 13,600 images, some of which achieve gigapixel. Intuitively, an image with more pixels does not necessarily present finer texture since the physical size of the corresponding painting may be large as well. To measure the richness of texture, the physical size of paintings is considered to calculate pixel per inch (PPI) for each image. Finally, we construct the training set consisting of 2,000 images and the testing set of 100 images with relatively higher PPI. Fig. 5 shows some examples of PaintHD, which contains abundant textures.
To further evaluate the generalization capacity of the proposed method, we also test on the CUFED5 [37] dataset, which is designed specifically for Ref-SR validation. There are 126 groups of samples. Each group consists of one HR image and four references with different levels of similarity to the HR image. For simplicity, we adopt the most similar reference for each HR image to construct the testing pairs. The images in CUFED5 are of much lower resolution, e.g., 500×300, as compared to the proposed PaintHD dataset.

Experimental Results
We conduct extensive experiments to validate the contributions of each component in our method. We demonstrate the effectiveness of our method by comparing with other state-of-the-art methods quantitatively and visually. We also provide more details, results, and analyses in supplementary material 1 .

Effect of Wavelet Texture Loss
The wavelet texture loss is imposed on the high-frequency band of the feature maps, rather than directly applying on raw features like SRNTT [37] and traditional style trans- fer [10]. Comparison between the wavelet texture loss and tradition texture loss is illustrated in Fig. 6. To highlight the difference, weights on texture losses during training are increased by 100 times as compared to the default setting in Section 3.4. Comparing Figs. 6(c) and 6(d), the result without wavelet is significantly affected by the texture/color from the reference (Fig. 6(b)), lost identity to the input content. By contrast, the result with wavelet still preserves similar texture and color to the ground truth ( Fig. 6(a)).

Effect of Degradation Loss
To demonstrate the effectiveness of our proposed degradation loss L deg , we use SRResNet [20] as the backbone with scaling factor 4×. The SRResNet baseline is trained by using only reconstruction loss L rec . Similar as Fig. 3, we further introduce L deg and L rec to train another SRResNet. We evaluate the performance of those two baselines and test them on four standard benchmark datasets: Set5 [1], Set14 [33], B100 [23], and Urban100 [14]. We show the PSNR (dB) and SSIM [28] values in Tab. 1. We can see that SRResNet with L deg obtains higher PSNR and SSIM values with obvious improvements. Such observations not only demonstrate the effectiveness of L deg , but also are consistent with our analyses in Section 3.3.

Quantitative Results
We compare our method with state-of-the-art SISR and Ref-SR methods 2 . The SISR methods are EDSR [22], RCAN [35], and SRGAN [20], where RCAN [35] achieved state-of-the-art performance in terms of PSNR (dB). Due to limited space, we only introduce the state-of-the-art Ref-SR method SRNTT [37] for comparison. However, most of those methods are not originally designed for very large scaling factors. Here, to make them suitable for 8× and 16× SR, we adopt them with some modifications. In 8× case, we use RCAN [35]    First, SISR methods would obtain higher PSNR and SSIM values than those of Ref-SR methods. This is reasonable because SISR methods mainly target to minimize MSE, which helps to pursue higher PSNR values. But, when the scaling factor goes to larger (e.g., 16×), the gap among SISR methods also becomes very smaller. It means that it would be difficult to distinguish the performance between different SISR methods by considering PSNR/SSIM.
Another interesting observation is that, in the group of Ref-SR, SRNTT [37] would achieve higher PSNR/SSIM values than our method. It's predictable, as RCAN [35] has achieved top PSNR performance and helps upscale the input by 2× or 4× firstly. When we train our network with reconstruction loss L rec only, denoted as 'Ours-L rec ', we can achieve comparable PSNR values as SRNTT. However, 'Ours-L rec ' achieves significant SSIM improvement, which means our method transfers more structural information.
Based on the observations and analyses above, we conclude that we should turn to other more visually-perceptual ways to evaluate the performance of SR methods, instead of only depending on PSNR/SSIM values. Consequently, we conduct more comparisons visual results and user study.  [35] and SRNTT [37]. We show visual comparisons in Figs. 7 and 8 for 8× and 16× cases respectively. In the 8× case, RCAN could handle it to some degree, because the LR input has abundant details for reconstruction. But, RCAN still suffer from some blurring artifacts due to use PSNR-oriented loss function (e.g., 1 -norm loss). By transferring textures from reference and using other loss functions (e.g., texture, perceptual, and adversarial losses), SRNTT [37] performs visually better than RCAN. But SRNTT still can hardly transfer more detailed textures. In contrast, our method would obviously address the blurring artifacts and can transfer more vivid textures.

Visual Comparisons
When the case goes more challenging, namely 16×, both RCAN and SRNTT would generate obvious oversmoothing artifacts (see Fig. 8). The reasons for RCAN are mainly that it aims to narrow the pixel-wise difference between the output I SR and the ground truth I GT . Such an optimization way would encourage more low-frequency components, but restrain the generation of high-frequency ones. SRNTT is originally designed for 4× upscaling, which restricts its ability for larger scaling factors. SRNTT neglects to pay more attention to the recovery of high-frequency components. In contrast, our method establishes trainable network from the original LR input to the target HR output firstly. Moreover, we focus on the reconstruction of high-frequency components more with wavelet texture loss. We even further relax the constraint between the output and ground truth by propose the degradation loss. As a result, our method alleviates the over-smoothing artifacts to some degree and recovers more detailed textures (see Fig. 8).

User Study
Since the traditional metric PSNR and SSIM do not consistent to visual quality [15,20,24,37], we conducted user study by following the setting of SRNTT [37] to compare our results with those from other methods, i.e., SRNTT [34], SRGAN [20], RCAN [35], and EDSR [22]. The EDSR and RCAN achieve state-of-the-art performance in terms of PSNR/SSIM, while SRGAN (SISR) and SRNTT (Ref-SR) focus more on visual quality. All methods are tested on a random subset of CUFED5 and PaintHD at the upscaling factor of 8× and 16×. In each query, the user is asked to select the visually better one between two side-by-side images super-resolved from the same LR input, i.e., one from ours and the other from another method. In total, we collected 3,480 votes, and the results are shown in Fig. 9. The height of a bar indicates the percentage of users who favor our results as compared to those from a corresponding method as denoted under the bar.
In general, our results achieve better visual quality at both upscaling scales, and the relative quality at 16× further outperforms the others. The main reason lies in the texture transfer from references. With the increase of the upscaling factor, more details are lost in the LR input, which is difficult to be recovered solely by the deep model. Thus, externally high-frequency information tends to be more important to texture recovery, which causes the gap from 5%

Conclusion
We aim to hallucinate painting images with very large upscaling factors and transfer high-resolution (HR) detailed textures from HR reference images. Such a task could be very challenging. The popular single image superresolution (SISR) could hardly transfer textures from reference images. On the other hand, reference-based SR (Ref-SR) could transfer textures to some degree, but could hardly handle very large scaling factors. We address this problem by first construct an efficient Ref-SR network, being suitable for very large scaling factor. To transfer more detailed textures, we propose a wavelet texture loss to focus on more high-frequency components. To alleviate the potential oversmoothing artifacts caused by reconstruction constraint, we further relax it by proposed a degradation loss. We collect high-quality painting dataset PaintHD, where we conduct extensive experiments and compare with other state-of-theart methods. We achieved significantly improvements over both SISR and Ref-SR methods. We believe such a Ref-SR network has promising benefits to general natural images.