{'fraction_non_alphanumeric': 0.03401799476646234, 'fraction_numerical': 0.012832921102627523, 'mean_word_length': 4.8096626405664304, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
NeRF-SR: High-Quality Neural Radiance Fields using Supersampling

We present NeRF-SR, a solution for high-resolution (HR) novel view synthesis with mostly low-resolution (LR) inputs. Our method is built upon Neural Radiance Fields (NeRF) that predicts per-point density and color with a multi-layer perceptron. While producing images at arbitrary scales, NeRF struggles with resolutions that go beyond observed images. Our key insight is that NeRF benefits from 3D consistency, which means an observed pixel absorbs information from nearby views. We first exploit it by a supersampling strategy that shoots multiple rays at each image pixel, which further enforces multi-view constraint at a sub-pixel level. Then, we show that NeRF-SR can further boost the performance of supersampling by a refinement network that leverages the estimated depth at hand to hallucinate details from related patches on only one HR reference image. Experiment results demonstrate that NeRF-SR generates high-quality results for novel view synthesis at HR on both synthetic and real-world datasets without any external information.


INTRODUCTION
Synthesizing photorealistic views from a novel viewpoint given a set of posed images, known as novel view synthesis, has been a long-standing problem in the computer vision community, and an important technique for VR and AR applications such as navigation, and telepresence. Traditional approaches mainly falls in the range of image-based rendering and follows the process of warping and blending source frames to target views [11,21]. Image-based rendering methods heavily rely on the quality of input data and only produces reasonable renderings with dense observed views and accurate proxy geometry.
(b) (a) Figure 1: NeRF, the state-of-the-art novel view synthesis method, can synthesize photorealistic outputs at the resolution of training images but struggles at higher resolutions as shown in (a), while NeRF-SR produces high-quality novel views (b) even with low-resolution inputs.
Most recently, neural rendering has made significant progress on novel view synthesis by leveraging learnable components with 3D geometry context to reconstruct novel views with respect to input images. As the current state-of-the-art method, neural radiance fields (NeRF) [32] have emerged as a promising direction for neural arXiv:2112.01759v3 [cs.CV] 21 Jul 2022 scene representation even on sparse image sets of complex realworld scenes. NeRF uses the weights of multi-layer perceptrons (MLPs) to encode the radiance field and volume density of a scene. Most importantly, the implicit neural representation is continuous, which enables NeRF to take as input any position in the volume at inference time and render images at any arbitrary resolution.
At the same time, a high-resolution 3D scene is essential for many real-world applications, e.g., a prerequisite to providing an immersive virtual environment in VR. However, a trained NeRF struggles to generalize directly to resolutions higher than that of the input images and generates blurry views (See Figure 1), which presents an obstacle for real-world scenarios, e.g., images collected from the Internet may be low-resolution. To tackle this problem, we present NeRF-SR, a technique that extends NeRF and creates high-resolution (HR) novel views with better quality even with low-resolution (LR) inputs. We first observe there is a sampling gap between the training and testing phase for super-resolving a 3D scene, since the sparse inputs are far from satisfying Nyquist view sampling rates [31]. To this end, we derive inspiration from traditional graphics pipeline and propose a supersampling strategy to better enforce the multi-view consistency embedded in NeRF in a sub-pixel manner, enabling the generation of both SR images and SR depth maps. Second, in the case of limited HR images such as panoramas and light field imaging systems that have a tradeoff between angular and spatial resolutions, we find that directly incorporating them in the NeRF training only improves renderings nearby the HR images in a small margin. Thus, we propose a patch-wise warp-and-refine strategy that utilizes the estimated 3D geometry and propagate the details of HR reference to all over the scene. Moreover, the refinement stage is efficient and introduces negligible running time compared with NeRF rendering.
To the best of our knowledge, we are the first to produce visually pleasing results for novel view synthesis under mainly lowresolution inputs. Our method requires only posed multi-view images of the target scene, from which we dig into the internal statistics and does not rely on any external priors. We show that NeRF-SR outperforms baselines that require LR-HR pairs for training.
Our contributions are summarized as follows: • the first framework that produces decent multi-view superresolution results with mostly LR input images • a supersampling strategy that exploits the view consistency in images and supervises NeRF in the sub-pixel manner • a refinement network that blends details from any HR reference by finding relevant patches with available depth maps 2 RELATED WORK Novel View Synthesis. Novel view synthesis can be categorized into image-based, learning-based, and geometry-based methods. Image-based methods warp and blend relevant patches in the observation frames to generate novel views based on measurements of quality [11,21]. Learning-based methods predict blending weights and view-dependent effects via neural networks and/or other handcrafted heuristics [5,12,41,53]. Deep learning has also facilitated methods that can predict novel views from a single image, but they often require a large amount of data for training [34,43,47,55,60].
Different from image-based and learning-based methods, geometrybased methods first reconstruct a 3D model [46] and render images from target poses. For example, Aliev et al. [1] assigned multiresolution features to point clouds and then performed neural rendering, Thies et al. [52] stored neural textures on 3D meshes and then render the novel view with traditional graphics pipeline. Other geometry representations include multi-planes images [9,22,23,31,50,68], voxel grids [13,17,39], depth [9,41,42,60] and layered depth [47,56]. These methods, although producing relatively highquality results, the discrete representations require abundant data and memory and the rendered resolutions are also limited by the accuracy of reconstructed geometry.
Together with differentiable rendering [18,27], these methods can reconstruct both geometry and appearance of objects and scenes [26,33,44,48,49]. Among them, Neural Radiance Fields (NeRF) [32] achieved remarkable results for synthesizing novel views of a static scene given a set of posed input images. There are a growing number of NeRF extensions emerged, e.g., reconstruction without input camera poses [25,59], modelling non-rigid scenes [28,36,37,40], unbounded scenes [64] and object categories [15,54,63]. Relevant to our work, Mip-NeRF [2] also considers the issue of resolution in NeRF. They showed that NeRFs rendered at various resolutions would introduce aliasing artifacts and resolved it by proposing an integrated positional encoding that featurize conical frustums instead of single points. Yet, Mip-NeRF only considers rendering with downsampled resolutions. To our knowledge, no prior work studies how to increase the resolution of NeRF.
Image Super-Resolution Our work is also related to image superresolution. Classical approaches in single-image super-resolution (SISR) utilize priors such as image statistics [19,69] or gradients [51]. CNN-based methods aim to learn the relationship between HR and LR images in CNN by minimizing the mean-square errors between SR images and ground truths [6,7,57]. Generative Adversarial Networks (GANs) [10] are also popular in super-resolution which hallucinates high resolution details by adversarial learning [20,29,45]. These methods mostly gain knowledge from large-scale datasets or existing HR and LR pairs for training. Besides, these 2D image-based methods, especially GAN-based methods do not take the view consistency into consideration and are sub-optimal for novel view synthesis. Reference-based image super-resolution (Ref-SR) upscales input images with additional reference high-resolution (HR) images. Existing methods match the correspondences between HR references and LR inputs with patch-match [66,67], feature extraction [61,62] or attention [62]. Although we also aim to learn HR details from given reference images, we work in the 3D geometry perspective and can bring details for all novel views instead of one image.

BACKGROUND
Neural Radiance Fields (NeRF) [32] encodes a 3D scene as a continuous function which takes as input 3D position x = ( , , ) and observed viewing direction d = ( , ), and predicts the radiance c(x, d) = ( , , ) and volume density (x). The color depends both on viewing direction d and x to capture view dependent effects, while the density only depends on x to maintain view consistency. NeRF is typically parametrized by a multilayer perceptron (MLP) : NeRF is an emission-only model (the color of a pixel only depends on the radiance along a ray with no other lighting factors). Therefore, according to volume rendering [16], the color along the camera ray r( ) = o + d that shots from the camera center o in direction d can be calculated as: where is the accumulated transmittance that indicates the probability that a ray travels from to without hitting any particle. NeRF is trained to minimize the mean-squared error (MSE) between the predicted renderings and the corresponding ground-truth color: where P denotes all pixels of training set images, r p ( ) = o + d p denotes the ray shooting from camera center to the corners (or centers in some variants [2]) of a given pixel p.Ĉ(r p ) and C(r p ) are the ground truth and output color of p.
In practice, the integral in Equation (1) is approximated by numeric quadrature that samples a finite number of points along with the rays and computes the summation of radiances according to the estimated per-point transmittance. The sampling in NeRF follows a coarse-to-fine mechanism with two MLPs, i.e., coarse network is queried on equally spaced samples whose outputs are utilized to sample another group of points for more accurate estimation and fine network is then queried on both groups of samples.

APPROACH
In this section, we introduce the details of NeRF-SR. The overall structure is presented in Figure 2. The supersampling strategy and patch refinement network will be introduced in Section 4.1 and Section 4.2.

Supersampling
NeRF optimizes a 3D radiance field by enforcing multi-view color consistency and samples rays based on camera poses and pixels locations in the training set. Although NeRF can be rendered at any resolution and retain great performance when the input images satisfy the Nyquist sampling rate, it is impossible in practice. Compared to the infinity possible incoming ray directions in the space, the sampling is quite sparse given limited input image observations. NeRF can create plausible novel views because the output resolution is the same as the input one and it relies on the interpolation property of neural networks. However, this becomes a problem when we render an image at a higher resolution than training images, specifically, there is a gap between the training and testing phase. Suppose a NeRF was trained on images of resolution H × W,  the most straightforward way to reconstruct a training image on scale factor , i.e., an image of resolution H × W is sampling a grid of 2 rays in an original pixel. Obviously, not only the sampled ray directions were never seen during training, but the pixel queried corresponds to a smaller region in the 3D space. Regarding this issue, we propose a supersampling strategy that tackles the problem of rendering SR images for NeRF. The intuition of supersampling is explained as follows and illustrated in Figure 3.
We start from the image formation process. The pixel values are mapped from scene irradiance through a camera response function (CRF). For simplification, we assume a pinhole camera model as in NeRF and consider ISO gain, shutter speed as implicit factors. Let R (p) denotes the set of all possible ray directions for pixel p from a training image, then: where C(p) indicates the color of p, is CRF, is the incident irradiance over the area covered by p, which is the integration of radiance over all incoming rays in p. Although ideally the training ray directions should be sampled from R (p), it is both computational expensive and challenging for the network to fit this huge amount of data. Therefore, in our work, to super-resolve images at the scale of , we first evenly split a pixel from training set into a × grid sub-pixels S(p). As in NeRF, we do not model CRF and output the color of each sub-pixel using a multi-layer perceptron directly. During training stage, ray directions for a pixel p will be sampled from the sub-pixels instead, denoted as R ′ (p) = {r j | j ∈ S(p)} ⊂ R (p). Figure 3: Original NeRF casts a single ray through a pixel (solid line) and performance MSE loss directly (left), while our method (right) splits a pixel into multiple sub-pixels (dash line) and draws a ray for each sub-pixel, then the radiances of sub-pixels will be averaged for MSE loss. Compared to vanilla NeRF, more 3D points in the scene can be corresponded and constrained in supersampling.

NeRF Ours
At inference stage, an H × W image can be directly obtained by directly rendering and organizing the sub-pixels, erasing the sampling gap between the training and testing phase. Another concern is how to perform supervision with only ground truth images at dimension H × W. Similar to the blind-SR problem, the degradation process from H × W is unknown and may be affected by many factors. Inspired by the graphics pipeline, we tackle this issue by compute the radiance for sub-pixels in R ′ ( ) using Equation 1 and then average them to compare with the color of p. Thus, Equation 3 can be extended as: where R ′ (p) is the sub-pixel grid for pixel p, |R ′ (p)| is the number of sub-pixels in R ′ (p), r ′ is the ray direction for a single sub-pixel, C(r ′ ) is the color of a sub-pixel predicted by the network. On the other hand, the LR images can be seen as downsampled from HR ones by averaging pixel color in a grid (We call it "average" kernel). This aborts any complex assumptions on the downsampling operation and make our method robust for various situations.
To summarize, supersampling extends original NeRF in two aspects: first it samples ray directions from × grid sub-pixels for pixel p instead of a single ray direction; second, it averages the color of the sub-pixels for supervision. In computer graphics, supersampling and averaging is often used in the rendering process to handle the problem of aliasing. In our work, we show that it fully exploits the cross-view consistency introduced by NeRF to a sub-pixel level, i.e., a position can be corresponded through multiple viewpoints. While NeRF only shoots one ray for each pixel and optimizes points along that ray, supersampling constraints more positions in the 3D space and better utilize the multi-view information in input images. In other words, supersampling directly optimizes a denser radiance field at training time.

Patch-Based Refinement
With supersampling, the synthesized image achieves much better visual quality than vanilla NeRF. However, when the images for a scene do not have enough sub-pixel correspondence, the results of supersampling cannot find enough details for high-resolution synthesis. Also, often there are limited high-resolution images from which HR content are available for further improving the results.
Here, we present a patch-based refinement network to recover high-frequency details that works even in the extreme case, i.e., only one HR reference is available, as shown in Figure 4. Our system is though not limited to one HR reference and can be easily extended to multiple HR settings. The core design consideration focuses on how to "blend" details on the reference image I REF into NeRF synthesized SR images that already captured the overall structure. We adopt a patch-by-patch refine strategy that turns an SR patch into the refined patch . Other than , the input should also include an HR patch from I REF that reveals how the objects or textures in presents in high-resolution. However, due to occlusion and inaccuracy of depth estimation, multiple HR patches are required to cover the region in and we use patches { REF } =1 for reference. Also, patches in { REF } =1 cover larger regions than and contain less relevant information. The refinement stage aims at local detail enhancement and well preserve the view consistent structure from super-sampling with the spatial information of depth predictions.
We use a U-Net based convolutional architecture for the refinement network, which has demonstrated its efficacy in several existing novel view synthesis methods [5,41,42]. In earlier attempts, we model the refinement procedure as an image-to-image translation [14] and find channel-wise stack and { REF } =1 were unable to fit the training set perfectly. Therefore, inspired by [5,41], we instead encode each patch respectively with an encoder consisting of seven convolutional layers. The decoder of the network takes as input the nearest-neighbor upsampled features from previous layers concatenated with both the encoded features of and maxpooled features of { REF } =1 at the same spatial resolution. All convolutional layers are followed by a ReLU activation.
Training The training of the refinement network requires SR and HR patch pairs, which are only available at the camera pose of I REF . Therefore, is randomly sampled from the SR image and is the patch on I REF at the same location. We perform perspective transformations to and as during testing, the input patches are mostly from different camera poses. Moreover, to account for the inaccuracy of reference patches at testing time, we sample { REF } =1 within a fixed window around . In order to preserve the spatial structure of while improving its quality, our objective function combines reconstruction loss L and perceptual loss L , where is a set of layers in a pretrained VGG-19 and is the reciprocal of the number of neurons in layer . Note that we adopt 1 -norm instead of MSE in because it is already minimized in supersampling and 1 -norm will sharpen the results.
Testing At inference time, given a patch on synthesized image I , we can find a high-resolution reference patch on reference image where , denotes a location on patch , is the estimated depth, is the transformation between camera extrinsic matrices from I to The training of NeRF requires correspondences of input images in the 3D space. As long as the HR reference falls in the camera frustum of input images, it can be easily wrapped to other views and bring enough details. Therefore, our refinement network is well-suited for any NeRF compatible dataset.

EXPERIMENTS
In this section, we provide both quantitative and qualitative comparisons to demonstrate the advantages of the proposed NeRF-SR. We first show results and analysis of super-sampling, and then demonstrate how the refinement network adds more details to it. Our result only with super-sampling is denoted as Ours-SS and our result after patch-based refinement is denoted as Ours-Refine.

Dataset and Metrics
To evaluate our methods, we train and test our model on the following datasets. We evaluate the quality of view synthesis with respect to ground truth from the same pose using three metrics: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) [58] and LPIPS [65].
Blender Dataset The Realistic Synthetic 360 • of [31] (known as Blender dataset) contains 8 detailed synthetic objects with 100 images taken from virtual cameras arranged on a hemisphere pointed inward. As in NeRF [32], for each scene we input 100 views for training and hold out 200 images for testing.
LLFF Dataset LLFF dataset [31,32] consists of 8 real-world scenes that contain mainly forward-facing images. We train on all the images and report the average metrics on the whole set.

Training Details
In super-sampling, we implement all experiments on top of NeRF [32] using PyTorch. As we train on different image resolutions independently, for fair comparison we train blender dataset and LLFF dataset for respectively 20 epochs and 30 epochs, where each epoch contains an iteration of the whole training set. We choose Adam as the optimizer (with hyperparameters 1 = 0.9, 2 = 0.999) with batch size set to 2048 (2048 rays a batch for all experimented scales) and learning rate decayed exponentially from 5 · 10 −4 to 5 · 10 −6 . Following NeRF, NeRF-SR also uses a hierarchical sampling with the same size "coarse" and "fine" MLP. The number of coarse samples and fine samples are both set to 64.

Comparisons
Since there are no previous work that deals with super-resolving NeRF, we devise several reasonable baselines for comparisons, detailed as the following: NeRF Vanilla NeRF is already capable of synthesising images at any resolution due to its implicit formation. Therefore, we train NeRF on LR inputs using the same hyperparameters in our method and directly render HR images.
NeRF-Bi aims to super-resolve a trained LR NeRF. We use the same trained model in the NeRF baseline, but render LR images directly and upsample them with the commonly used bicubic upsampling.
NeRF-Liif Liif [3] achieves state-of-the-art performance on continuous single image super-resolution. Similar to the NeRF-Bi baseline, we super-resolve LR images using pretrained liif model instead. Note that to the training process of liff requires LR-HR pairs, therefore it introduces external data priors.    performs super-resolution on a LR NeRF with the released SwinIR models under the "Real-World Image Super-Resolution" setting, which has a training set of more than 10k LR-HR pairs.

Effectiveness of supersampling
For blender dataset, we super-sample on two resolutions: 100 × 100 and 200 × 200, and test scales ×2 and ×4. For the LLFF dataset, the input resolution is 504 × 378 and we also upscale by ×2 and ×4. The downscaling of images in the dataset from original resolution to training resolution is done by the default Lanczos method in the Pillow package. Figure 5 shows qualitative results for all methods on a subset of blender scenes. Renderings from NeRF-Bi exhibit correct global shapes but lack high-frequency details. Vanilla NeRF produces renderings that have more details than NeRF-Bi if the scene is already well-reconstructed at input resolution. However, it is still restricted by the information in the input image. NeRF-Liif can recover some details, but lacks enough texture. NeRF-SR find sub-pixel level correspondence through supersampling, which means missing details in the input can be found from other views that lie in the neighboring region in 3D space.
Quantitative results of blender dataset are summarized in Table 1. NeRF-SR outperforms other baselines in all scenarios. NeRF-Liif or NeRF-Swin have the second best LPIPS, providing good visual quality but cannot even compete with NeRF in PSNR and SSIM. The reason is maybe the blender dataset is synthetic and has different domain than the dataset it is trained on, resulting false prediction (see NeRF-Swin on Lego and Hotdog).
The qualitative and quantitative results for LLFF dataset are demonstrated in Figure 6 and Table 2 respectively. NeRF and NeRF-Bi suffers from blurry outputs. While NeRF-Liif and NeRF-Swin recovers some details and achieve satisfying visual quality (comparable LPIPS to Ours-SS) since they are trained on external datasets, they tend to be oversmooth and even predicts false color or geometry (See the leaves of Flower in Figure 6). NeRF-SR fill in the details on the complex scenes and outperforms other baselines significantly. Therefore, we can conclude that learning-based 2D baselines struggle to perform faithfully super-resolution, especially in the multi-view case.
In Section 4.1, we mentioned that the supervision is performed by comparing the average color of sub-pixels due to the unknown nature of the degradation process (We call it "average kernel"). However, in our experiments, the degradation kernel is actually Lanczos, resulting an asymmetric downscale and upscale operation. We further experiment on the condition that the degradation from high-resolution to input images is also "average kernel" for blender data at the resolution 100 × 100. Results show this symmetric downscale and upscale operation provides better renderings than asymmetric one. PSNR, SSIM, LPIPs are all improved to 30.94 dB, 0.956, 0.023 for scale ×2 and 28.28 dB, 0.925 and 0.061 for ×4 respectively. The sensitivity to the degradation process is similar to that exhibited in single-image super-resolution. Detailed Rendering can be found in the supplementary.

Refinement network
LLFF dataset contains real-world pictures that have a much more complex structure than the blender dataset, and super-sampling isn't enough for photorealistic renderings. We further boost its outputs with a refinement network introduced in Section 4.2. We use a fixed number of reference patches ( = 8) and the dimensions of patches are set to 64 × 64. While inferencing, the input images are divided into non-overlapping patches and stitched together after refinement. Without the loss of generosity, we set the reference image is to the first image in the dataset for all scenes, which is omitted when calculating the metrics. The inference time of the refinement stage is neglibile compared to NeRF's volumetric rendering: for example, it takes about 48 seconds for NeRF's MLP to render a 1008 × 756 image, and it only takes another 1.3 seconds in the refinement stage on a single 1080Ti.
The quantitative results of refinement can be found in Table 2. After refinement, metrics are improved substantially at the scale of 4. For the scale of 2, PSNR increases only a bit after refining, a possible reason is that supersampling already learns a decent high-resolution neural radiance fields for small upscale factors and the refinement only improves subtle details (Please refer the supplementary for an example). However, we can see that LPIPS is still promoted, meaning the visual appearance improves. The problem doesn't occur for larger magnifications such as 4 since supersampling derives much fewer details from low-resolution inputs, making the refinement process necessary.
We demonstrate the renderings qualitatively before and after refining in Figure 6. It is clear to see that the refinement network boosts supersampling with texture details and edge sharpness.

LIMITATIONS AND CONCLUSION
A major limitation of NeRF-SR is that it does not enjoy the nice arbitrary-scale property. It also introduces extra computation efficiency, albeit it consumes no more time than training a HR NeRF.
In conclusion, we presented NeRF-SR the first pipeline of HR novel view synthesis with mostly low resolution inputs and achieve photorealistic renderings without any external data. Specifically, we exploit the 3D consistency in NeRF from two perspectives: supersampling strategy that finds corresponding points through multi-views in sub-pixels and depth-guided refinement that hallucinates details from relevant patches on an HR reference image. Finally, region sensitive supersampling and generalized NeRF superresolution may be explored for future works.

ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (Project Number 62132012).