{'fraction_non_alphanumeric': 0.032218173788230095, 'fraction_numerical': 0.012233631445290934, 'mean_word_length': 4.702808210552827, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations

A classical problem in computer vision is to infer a 3D scene representation from few images that can be used to render novel views at interactive rates. Previous work focuses on reconstructing pre-defined 3D representations, e.g. textured meshes, or implicit representations, e.g. radiance fields, and often requires input images with precise camera poses and long processing times for each novel scene. In this work, we propose the Scene Representation Transformer (SRT), a method which processes posed or unposed RGB images of a new area, infers a"set-latent scene representation", and synthesises novel views, all in a single feed-forward pass. To calculate the scene representation, we propose a generalization of the Vision Transformer to sets of images, enabling global information integration, and hence 3D reasoning. An efficient decoder transformer parameterizes the light field by attending into the scene representation to render novel views. Learning is supervised end-to-end by minimizing a novel-view reconstruction error. We show that this method outperforms recent baselines in terms of PSNR and speed on synthetic datasets, including a new dataset created for the paper. Further, we demonstrate that SRT scales to support interactive visualization and semantic segmentation of real-world outdoor environments using Street View imagery.

In this work, we propose the Scene Representation Transformer (SRT), a method which processes posed or unposed RGB images of a new area, infers a "set-latent scene representation", and synthesises novel views, all in a single feed-forward pass. To calculate the scene representation, we propose a generalization of the Vision Transformer to sets of images, enabling global information integration, and hence 3D reasoning. An efficient decoder transformer parameterizes the light field by attending into the scene representation to render novel views. Learning is supervised end-to-end by minimizing a novel-view reconstruction error.
We show that this method outperforms recent baselines in terms of PSNR and speed on synthetic datasets, including a new dataset created for the paper. Further, we demonstrate that SRT scales to support interactive visualization and semantic segmentation of real-world outdoor environments using Street View imagery.

Introduction
The goal of our work is interactive novel view synthesis: given few RGB images of a previously unseen scene, we synthesize novel views of the same scene at interactive rates and without expensive per-scene processing. Such a system can be useful for virtual exploration of urban spaces [26], as well as other mapping, visualization, and AR/VR applications [27]. The main challenge here is to infer a scene representation that encodes enough 3D information to render novel views with correct parallax and occlusions.
More recent work has investigated representing scenes with purely implicit representations [39]. Notably, NeRF represents the scene as a 3D volume parameterized by an MLP [25], and has been demonstrated to scale to challenging real-world settings [23]. However, it typically requires hundreds of MLP evaluations for the volumetric rendering of each ray, relies on accurate camera poses, and requires an expensive training procedure to onboard new scenes, as no parameters are shared across different scenes.
Follow-ups that address these shortcomings include reprojection based methods which still rely on accurate camera poses due to their explicit use of geometry, and latent models which are usually geometry-free and able to reason globally, an advantage for the sparse input view setting. However, those methods generally fail to scale to complex real-world datasets. As shown in Tab. 1, interactive novel view synthesis on complex real-world data remains a challenge.
We tackle this challenge by employing an encoderdecoder model built on transformers, learning a scalable implicit representation, and replacing explicit geometric operations with learned attention mechanisms. Different from a number of prior works that learn the latent scene representations through an auto-decoder optimization procedure [38,16], we specifically rely on an encoder architecture to allow for instant inference on novel scenes. At the same time, we do not rely on explicit locally-conditioned geometry [49,41] to instead allow the model to reason globally. This not only comes with the advantage of stronger generalization abilities (e.g., for rendering novel cameras further away from the input views, see Fig. 4), but also enables the model to be more efficient, as global information is processed once per scene (e.g., to reconstruct 3D geometry and to resolve occlusions) in our model, rather than once or hundreds of times per rendered pixel, as in previous methods [49,41,44].
We evaluate the proposed model on several datasets of increasing complexity against relevant prior art, and see that it strikes a unique balance between scalability to complex scenes (Sec. 4), robustness to noisy camera poses (or no poses at all, Fig. 6), and efficiency in interactive applications (Tab. 3). Further, we show a proof-of-concept for the downstream task of semantic segmentation using the learned representation on a challenging real-world dataset (Fig. 7).

Related Work
There is a long history of prior work on novel view synthesis from multiple RGB images based on neural representations (see Tab. 1 and [40]). Recent work has demonstrated that novel views can be synthesized using deep networks trained to compute radiance values for a given 3D position and direction. In particular, NeRF [25] trains an MLP to store radiance and density used in a volume rendering system and produces novel views with remarkably fine details. Several extensions have been proposed to mitigate some its shortcomings, including ones to accelerate rendering [12,30,48], handle color variations [23], optimize camera poses [21], and perform well with few images [15]. However, these methods generally still require an expensive optimization of an MLP for each novel scene, and thus are impractical for deployment at large scale. NeRF with Re-projection. Several NeRF extensions have been proposed that do not always require to be trained indi-NeRF et al.

Re-projection Latent Other Ours
NeRF [25] FastNeRF [9] NeRF-W [23] BARF [21] DietNeRF [15] GRF [41] IBRNet [44] PixelNeRF [49] MVSNeRF [5] CodeNeRF [16] NeRF-VAE [20] LFN [38] NRW [24] GFVS [32] SRT vidually for each novel scene. These methods rely on explicit 3D-to-2D projections of samples along the rays to gather features extracted from CNNs, which are then aggregated by a neural network for each point [41,44,49], or into a voxel grid [5]. As a result, these networks can be pre-trained, and then applied to novel scenes either without any optimization, or with additional per-scene fine-tuning. However, reliance on explicit camera projections requires precise camera poses, and direct projections cannot take occlusions of objects between samples and input features into account. Light Fields. Other methods learn to represent a scene with a latent vector from which novel views can be synthesized [7]. More recently, LFN [38] use an auto-decoder to infer a latent code from input images, which then conditions an MLP for computing radiance values for rays of novel views. This method is several times faster than volumetric rendering due to the fact that the network is only queried once per pixel, rather than hundreds of times for volumetric rendering. However, it does not scale beyond simple controlled settings, requires an expensive optimization to produce a latent code for a novel scene, and therefore also relies upon accurate input camera poses. In contrast, we utilize an encoder to infer the scene representation, which executes nearly instantly in comparison (see Tab. 3), and can leverage, but does not require, camera poses (see Sec. 4.4).
Transformers. Originally proposed for seq-to-seq tasks in natural language processing, transformers [43] drop the recurrent memory of prior methods, and instead use a multihead attention mechanism to gather information from the most relevant portions of the input. The vision community has deeply researched applications thereof in image  and video processing, where they have meanwhile achieved state of the art in several tasks [18], including inherently geometric applications such as depth estimation [29], point cloud processing [51], and generative geometry-free view synthesis [32]. The latter work uses transformers to learn a sequence of VQ-GAN embeddings [8] to sample realistic views given a single image. While results are of high-quality, this method is slow due to its auto-regressive nature, and cannot be used for video rendering, as each frame is sampled independently, creating strong flickering artifacts. ViT [6] has been proposed for very large-scale classification tasks: local features are extracted from patches of the input image, and fed to a transformer network. We will demonstrate that this architecture is well-suited for learning 3D scene representations by building on top of this idea, extending it to ingest patches from multiple images, and generalizing it to 3D by adding a transformer decoder queried with rays.

Method
Our model receives as input an unordered collection of optionally posed images {I i ∈ R H×W ×3 , c i ∈ SE(3), K i ∈ R 3×3 } of the same scene, with extrinsic camera matrix c i = [R i |t i ] and intrinsic camera matrix K i . The model encodes this collection into a set representation of cardinality Z via an encoder E with trainable parameters θ E : In more details, our encoder first processes the images via a shared CNN backbone (Fig. 2, left), and then aggregates features into a set of flat patch embeddings. This set of embeddings is then processed by the encoder transformer that generates the codes {z z } as output. Note that {z z } fully encodes a specific 3D scene as observed by the corresponding set of images, hence it is our set-latent scene representation.
This representation is queried by the decoder transformer D with trainable parameters θ D to generate the pixel color for a view ray r = (o, d) with origin o and direction d: By convention, we assume I 0 to be the canonical image, hence queries r are expressed in the coordinates of c 0 , and, whenever camera information is not available, we assume the canonical camera pose to be identity. If pose information is known, images within a scene are randomly shuffled, hence the reference frame is randomly chosen both at train and test time. Given a collection of images {I s,i } from different scenes indexed by s, we train all parameters end-to-end via a reconstruction loss on novel-view synthesis: As both inputs and outputs of the model during training are 2D images, there is no explicit 3D supervision. We now describe the internals of our architecture in details, as illustrated in Fig. 2.
Pose information. We start by concatenating along the channel dimension the positional encoding γ [43,25]  L d octaves, as seen from the currently selected canonical camera c 0 to produce: Overall, the output of this stage will have dimensional- , depending on whether camera pose information is provided or not. Following [25], we perform sine and cosine encoding independently for each of the 3D dimensions (i.e., 6L). CNN - Fig. 2 (left). A shared convolutional neural network extracts patch features while scaling the spatial dimensions of the input images by a factor of K=16, hence mapping the shape from (H, W, C) to (h = H /K, w = W /K, C): We add a single globally learned 2D position embedding B(θ) ∈ R h×w×C to all patches of all images, to allow the model to retain the position of each patch in the corresponding image. The spatial dimensions are then flattened to obtainF CNN i ∈ R hw,C . Further, one of two learned camera id embeddings B i=0 (θ), B i =0 (θ) ∈ R C are added to all patches of the canonical camera, and all remaining cameras, respectively. This is to allow the model to distinguish the reference camera from all others: ., hw} (7) All features together form (unsorted) sets of patch embeddings of cardinality |{f f }| = Ihw, each of size R C : Encoder Transformer - Fig. 2 (center). We pass the patch embeddings through a standard transformer [43] that alternates between attention to all tokens, and small MLP blocks: This network integrates information of the scene at a global scale by attending across patches and input images to infer a 3D scene representation. Note that crucially, the set-latent  Table 3. Computational performance -Model size, and a) time required to encode a scene, b) frame rate for individually rendering 100 frames after encoding, c) application: total time to encode a novel scene and render a video of 100 frames. SRT is orders of magnitudes faster for real-world use. See Appendix A.2 for details.
scene representation scales in size with the amount of input information rather than being fixed-size [16,20,38]. Decoder Transformer - Fig. 2 (right). The decoder D is also a transformer. The main difference with respect to the encoder is that the ray corresponding to the pixel to be rendered is used as the query for the multi-head attention mechanism, while key and value are computed from the scene representation {z z } in all layers. In other words, the decoder learns to attend to the most relevant subset of features in the scene representation to calculate the output color. Note that the query ray position is encoded as in Eq. (4). The output of the decoder is finally passed to a small 2-layer MLP to compute the pixel color.

Training and Inference
During training, all components are simply trained endto-end using an L2 reconstruction loss on novel views, see Eq. (3). We provide further training details in the Appendix A.2. For inference, input images are encoded into a scene representation once, which can then be used to render an arbitrary number of novel views (e.g., for a video). Note that this methodology is designed to maximize inference efficiency, as the 3D reasoning is executed only once in the encoder transformer, independent of the number of novel views to be rendered. For downstream tasks, the encoder for the scene representation can be pre-trained and either frozen, or fine-tuned along with a new decoder for a novel task (e.g. semantic segmentation -Sec. 4.5).

Experimental Results
We run a series of experiments to evaluate how well SRT performs novel view synthesis in comparison to previous works and to study how each component of the network architecture contributes to the results. In all experiments, all methods are pre-trained on a dataset of images from one set of scenes and then tested on images from novel scenes. Unless stated otherwise, we train SRT with the same model architecture, hyper parameters, and training protocols for all experiments across all datasets. Synthesized images are evaluated with PSNR, SSIM [46], and LPIPS [50] (VGG).

Datasets
Neural 3D Mesh Renderer Dataset (NMR) [17]. NMR has been used in several previous studies. It consists of ShapeNet [4] objects rendered at 24 fixed views. The dataset is very simple, and likely does not provide good evidence for the applicability in real-world settings. MultiShapeNet (MSN). We therefore propose this significantly more challenging dataset rendered using photorealistic ray tracing [11]. Each scene has 16-31 ShapeNet objects dropped at random locations within an invisible bounding box. We further sample from 382 complex HDR backgrounds and environment maps. Viewpoints are selected by uniformly sampling within a half-sphere shell.
This dataset is significantly more difficult than NMR for several reasons: 1) images are rendered photorealistically, 2) scenes contain many different types of objects in complex, tightly-packed configurations, 3) the nontrivial background maps hamper the model's abilities to segment the objects (just like in the real world), and 4) the randomly sampled viewpoints prevent models from overfitting, unlike NMR where the same cameras and canonically oriented objects are used. The dataset is available at srt-paper.github.io. Street View. This is a new dataset created from real-world Street View data [10]. The training dataset contains 5.5 M scenes from San Francisco, each with 10 viewpoints. This dataset is very challenging, since the input viewpoints are typically several meters from the reference viewpoints, the cameras are often arranged in straight lines (along the car trajectory, meaning there is not a lot of diversity in viewing angles), the scenes contain challenging geometry (e.g., trees and thin poles) and dynamic elements (e.g., moving vehicles and pedestrians), the cameras have fisheye distortion and rolling shutter (making reprojection-based methods such as [49,41,44] inapplicable), and finally, different images often contain different exposure and white-balance settings.

Comparisons to Baselines
In the first set of experiments, we compare the results of SRT to previous methods for novel view synthesis. For these experiments, we restrict our tests to the NMR and MSN datasets, since they provide an opportunity for direct comparisons to recently published methods in a synthetic setting with known ground truth. We run SRT with the same parameters on both datasets. Baselines. We provide comparisons to two baselines that share aspects of our approach. The first baseline is Pixel-NeRF [49], which also pretrains a network to produce features from images. The second baseline is Light Field Networks (LFN) [38], which like our method uses a light field formulation rather than volumetric rendering. These methods have demonstrated favorable quantitative and qualitative results in comparison to other previous work [41,37,28,22].
Input LFN [38] PN [49] SRT (Ours) Target Figure 3. Qualitative NMR results -While PixelNeRF has highquality for target views close to the input (middle), results quickly degrade when further away (bottom). The quality of SRT renders is much more consistent and outperforms both baselines. More results are available at srt-paper.github.io.
Quantitative Results -Tabs. 2 and 3. SRT provides the best image quality on NMR, outperforming LFN and Pix-elNeRF. In terms of computational performance (Tab. 3), both PixelNeRF and SRT are fast to encode new scenes, while LFN requires slow optimization of the scene embedding. Once a scene is encoded, both LFN and SRT render new frames at interactive rates, while PixelNeRF is significantly slower due to the volumetric rendering. SRT is the only method fast at both scene encoding and novel-view generation, making it suitable for the practical application of rendering videos of novel scenes at interactive rates. See Appendix A.2 further details. We note that SRT is comparably small for a transformer: its 74 M parameters consist of 23 M for CNN, 47 M for Encoder, and 4 M for Decoder. For comparison, ViT [6] has 86 -632 M parameters.
On the MSN dataset, SRT outperforms LFN and Pix-elNeRF in terms of PSNR and SSIM. In terms of LPIPS, PixelNeRF outperforms SRT, which we attribute to the fact that SRT tends to blur regions of uncertainty. In a separate experiment, we tried increasing the latent dimension of LFN (from 256 to 1024) to account for more complex scenes, but this did not increase PSNR on novel scenes.
Qualitative Results - Figs. 3 and 4. The results show that our method outperforms both baselines on the NMR dataset. While all methods are able to synthesize high quality images for nearby views (Fig. 3, center), PixelNeRF renders degrade particularly quickly as the camera moves away from the input views (Fig. 3, bottom). LFN render quality degrades similarly, and it is also blurry for more complex objects. In contrast, SRT provides similar performance for a wide range of views. The MSN dataset (Fig. 4) with its complex scenes clearly demonstrates the limits of LFN's single-latent scene representation. While PixelNeRF results in detailed reconstructions for target views close to the inputs (Fig. 4 middle), SRT is the only method that allows for good reconstructions of far-away target views (Fig. 4 bottom).
Model Inspection - Fig. 5  by investigating the attention weights of the encoder and decoder transformers. We can see from Fig. 5 (top) that the patch with the small chair (marked with a green box) mostly attends into the same chair in the other input images, even when the chair is facing different directions. We further noticed that all patches additionally attend into a subset of patches positioned along the edges of the images (bottom corners of the first input image). We believe that the encoder learns to store global information in those specific patches. A glance at the decoder supports this conjecture. When rendering a ray of the chair from a novel direction, the first attention layer only attends into the bottom edge of all input views, while the second layer attends into similar patches as the encoder, and into the bottom corners of the first input image. This learned 2-stage inference in the decoder appears to be crucial, as we noticed in prior experiments that a 1-layer decoder is significantly worse, while more than two layers do not lead to significant gains. The model appears to have learned a hybrid global / local conditioning pattern through back propagation, without explicit geometric projections.

Ablation Studies
We now perform a series of ablation studies on the MSN dataset, where we remove or substitute the main components of SRT and measure the change in performance to evaluate individual contributions of different design decisions. No Encoder Transformer. Retraining the system without an encoder transformer (i.e., the decoder attends directly into the output of the CNN) provides a significant drop from 23.41 to 21.64 PSNR. This suggests that the encoder transformer adds crucial capacity for inference, and that the choice to move compute from the decoder (which should be as small and fast as possible) to the encoder is a feasi-ble design decision. Note that its computational overhead is negligible in most practical settings, as it is only a single feed-forward step, and only run once per novel scene, independent of the number of rendered frames. Flat Latent Scene Representation. One of the main novelties of SRT is the set-latent scene representation. We investigate the more commonly used flat latent by feeding the mean patch embedding of the encoder transformer into a large 8-layer MLP and dropping the decoder transformer. This architecture leads to a significant drop from 23.41 down to 20.88 in PSNR, showing the strength of a large set scene representation along with an attention-decoder. Volumetric Rendering. As an alternative to the light field formulation, we also investigate a volumetric parametrization. To this end, we simply query the decoder not with rays, but rather with 3D points, followed by the volumetric rendering [25]. For simplicity, we do not inject viewing directions, and only use a single coarse network. We call this variation V-SRT. While this variation leads to an explicit 3D volume with easy-to-visualize depth maps, and results are visually comparable to SRT, it is to be mentioned that the V-SRT's decoding is slower by a theoretical factor of 192×, the number of samples per ray, making inference time of this variation more similar to existing volumetric methods [49]. It is notable that our model architecture can be trained in both setups without further changes to architecture or hyper-parameters. See details in Appendix A.3.

Robustness Study
While perfect pose information is available in synthetic setups, real-world applications often depend on estimated camera poses (e.g., [35]), which is slow and often contains errors [21]. We therefore study SRT and baselines in a noisy camera pose regime. To this end, we follow [21] and synthetically perturb all input camera poses c i for i = 0 (all but the reference camera, see Sec. 3) with additive noise δc i = N (0, σ) to various degrees. For each value of σ, we retrain all models from scratch to allow them to adjust to the noise. To allow PixelNeRF to distinguish the reference input camera c 0 from the others, we add learned camera identity embeddings to the output of the CNN, see Sec. 3. PixelNeRF heavily relues on accurate camera poses to perform projections from 3D volumes into 2D images during rendering. As expected, Fig. 6 shows that the performance of this method degrades sharply even with small amounts of noise. LFN does not scale to the MSN dataset (see Fig. 4), and we note that minor amounts of noise actually increase PSNR due to a regularization effect. As LFN depends on test-time optimization for the scene latent, it is not robust to noisy camera positions.
In contrast, SRT handles noisy poses much more gracefully. Taking it to the extreme, our method even works with fully un-posed imagery, still outperforming both baselines. We call this model variation without test-time poses UpSRT. Further inspection shows that UpSRT is not only trivially using the first input image (for which the pose is known by definition, see Sec. 3) but the model is in fact learning to use all input cameras, even the un-posed ones. This is evidenced by inspection of attention patterns, and by the fact that UpSRT significantly outperforms SRT with a single input view only, see Appendix A.3.

Applications
In this section, we investigate the use of SRT in applications. For these experiments, we use the Street View dataset, which contains images of real-world outdoor environments.  Figure 6. Robustness -PixelNeRF [49] quality drops quickly even for minor amounts of noise despite being trained for this setting. LFN [38] fails to scale to MSN even with perfect cameras -some noise regularizes the model and actually leads to a small increase in quality. SRT handles camera noise gracefully, outperforming competing methods even without any camera parameters (UpSRT). present in this dataset, we augment the network with an appearance encoder, see Appendix A.2 for details.
View Interpolation. The goal in this application is to interpolate images captured in Street View panoramas to provide smooth video transitions between them. This application requires not only accurate view synthesis, but also temporal coherence between nearby views. We show a representative sample of interpolated views in Fig. 7. Although the renders are blurrier than the input views, our results demonstrate that SRT scales to complex real-world scenes with nontrivial camera pose distributions. It also shows that the model learns enough 3D scene information to render novel views far away from the inputs (last column). We provide videos and further results showing temporal coherence in render videos in Appendix A.3.
Semantic Segmentation. Our goal in this application is to predict dense semantics for novel views of outdoor scenes. Rather than synthesizing the color images for novel views and then employing a 2D image segmentation network on them, we show that SRT's scene representation trained on RGB can be leveraged more directly in new domains. Once SRT has been trained for the RGB reconstruction task, we freeze the encoder, and train a new decoder transformer to synthesize semantic segmentation images from the frozen scene representation directly. The semantic decoder has the same network architecture as the color decoder, except for the final output layer that is changed from 3 RGB channels to 46 semantic classes. We train the semantic decoder using the standard multi-class cross-entropy loss in a semi-supervised setup, see Appendix A.2.
Example semantic segmentation results are shown in Fig. 7. These results suggest that the scene representation learned through a color reconstruction loss contains enough information about the scene to allow semantic reasoning.

Nearby SRT Predictions
Far SRT Predictions Figure 7. Qualitative results on Street View -SRT performs reasonably on highly challenging real-world data with small and large changes in camera perspective. Furthermore, the scene representation contains enough information for 3D semantic scene inference. A comparison with NeRF-based optimization methods is provided in Appendix A.3.
Though this is just one example of many possible downstream tasks, it is a significant finding that SRT has learned a scene representation useful for a non-trivial application.

Limitations
This project investigates whether transformers can learn scalable scene representations from only images without explicit geometry processing. We here identify a few limitations, which we believe to be tackled by follow up works.
First, our results show some blurriness in images on the complex datasets, even for views close to the input cameras. This is expected, as the model needs to learn pixel-accurate light ray transformations, which leads to uncertainty about exact positions, known to result in blurriness under an L2 loss [33].
Second, SRT is a geometry-free learning-based method, and hence will not work as well on very small datasets compared to methods with explicit geometric inductive biases. In practice, we find that our model converges at rates similar to others, but even more training usually results in better performance. In particular, when trained on the standard datasets reported in this paper, SRT outperforms LFN and PixelNeRF. Further investigation is required to better understand the best training protocols.
Finally, in comparisons to prior art, our model is best when input views are sparse or when camera poses are noisy or missing. When novel views are reliably close to input views, and perfect poses are known, explicit geometric meth-ods such as [49] often provide better results, albeit with much longer inference times. While SRT has been specifically designed for the sparse input scenario, future work could investigate how to better leverage nearby input views at inference time when they are available.

Conclusion
We propose Scene Representation Transformer, a model for learning scalable neural scene representations using only self-supervision from color images. The novel encoderdecoder transformer architecture learns to render novel views without explicit geometric reasoning, and optionally without 3D image poses, yet surpassing the quality of prior art on standard benchmarks and a novel, more challenging dataset that we propose. At the same time, SRT is orders of magnitudes faster than prior methods for the realistic settings such as novel-scene video generation. This unique combination of flexibility, generality, and efficiency is well-suited for real world applications with very large datasets, and we believe that this work will inspire the community towards similar more implicit, yet scalable methods.

A. Appendix
We provide a discussion of potential negative societal impact and include further details and results. Video results are available on the project website at srt-paper.github.io.

A.1. Potential Negative Societal Impact
Fakes. This paper proposes a method to synthesize novel views of scenes observed from a small set of input views. As in any image synthesis system, it could be used to produce images that have misleading or fake content, particularly if an adversary selected the input images. We plan to mitigate this issue partially by deploying the algorithms only in secure servers when running them on real world datasets, and we will clearly mark all synthesized imagery as such to reduce the chance of misinterpretation.
Privacy. This paper includes experiments run with a Street View dataset containing images acquired in public areas of real world outdoor scenes. As in any image dataset, these images could have information that negatively affects personal privacy. To mitigate these potential impacts, we follow strict privacy policies required by the owner of the data, Google, which include rules for using the data in accordance with regional laws, anonymizing the data to avoid identification of people or license plates, and retaining the data in secure storage (https://policies.google.com/privacy).
Fairness. Our Street View dataset contains images captured by a car-mounted camera platform driving on public roads in San Francisco. Long sequences of images are captured systematically with fixed view directions and regular time intervals as the car drives. There are biases in the dataset based on which roads are driven in which city, which align with demographic or economic factors. However, individual images are not selected by a human photographer, which might mitigate some types of bias common in other computer vision datasets.
Energy consumption. The paper proposes a method to train a transformer network architecture to produce a scene representation and arbitrary novel images from a set of input images. The training procedure requires a lot of compute cycles, and thus has a negative impact on the environment due to its energy consumption. However, in comparison to most competing methods based on volumetric rendering methods, the inference cost is smaller (by 6-7 orders of magnitude per scene and 2-3 orders of magnitude per image). Thus, our proposed system is the most efficient of its kind when deployed at scale (our target use case) -where the system is trained on millions of images and then run without further fine-tuning to synthesize up to trillions of novel images.

A.2. Architecture, Training and Evaluation Dataset Details
Neural 3D Mesh Renderer Dataset (NMR) [17]. This dataset consists of ShapeNet [4] objects from the 13 largest classes, each with 24 fixed views rendered at 64×64 pixels. We use the SoftRas split [14], which contains 31 k training, 4 k validation and 9 k test objects. Following the protocol in [38,49], one randomly sampled image is provided as input, and images are rendered for all other 23 views. MultiShapeNet (MSN). We use 1 M scenes for training, and 10 k for testing. To ensure that all test set scenes contain novel objects, we split the full set of ShapeNet objects into a train and test split, and sample objects for each from the corresponding split. Viewpoints are selected by independently and uniformly sampling 10 cameras within a half-sphere shell of radius 8-12 for each scene, with all cameras pointed at the origin. All images are rendered at a resolution of 128×128. The task is to render novel views given 5 randomly selected input images of a novel scene. Street View. The resolution of this dataset is 176×128 pixels. During training, the task is to predict 5 randomly sampled images from the other 5 input views. At test time, we render views from novel viewpoints. To ensure that the evaluation scenes are new to the model, we physically separate them from the training scenes.

SRT Model Details
We use the same network architecture and hyper parameters throughout all experiments (with the exception of the appearance encoding module, see Sec. A.2). Our CNN consists of 4 blocks, each with 2 convolutional layers. The first convolution in each block has stride 1, the second has stride 2. We begin with 96 channels which we double with every strided convolution, and map the final activations with a 1×1 convolution (i.e., a per-patch linear layer) to 768 channels.
Both encoder and decoder transformers are similar to ViT-Base [6]: all attention layers have 12 heads, each with 64 channels, and the MLPs have 1 hidden layer with 1536 channels and GELU [13] activations. The final MLP for the decoder also has a single hidden layer, but with 128 channels, with a final sigmoid operation for the RGB output. For the positional encoding of ray origin and direction, we We train our model with the Adam optimizer [19] with initial learning rate 1 × 10 −4 that is smoothly decayed down to 1.6 × 10 −5 over 4 M training steps and a learning rate warmup phase in the first 2.5 k steps. We train with a batch size of 256 and for every data point, we randomly sample 8192 points uniformly across all target views for rendering and the loss computation. We noticed that the model was still not fully converged at 4 M training steps, and validation Target frame SRT V-SRT Figure 8. Epipolar Images (EPI) -For a dolly camera motion, the same horizontal line of pixels marked in green (left) is stacked vertically from all video frames, producing an EPI. The slope of each diagonal line in an EPI corresponds to the depth of that pixel in the scene. V-SRT uses a volumetric parametrization of the scene and no viewing directions. We can see that its EPI looks quite similar to SRT's, indicating that the latter's scene representation is geometrically and temporally stable, despite a lack of theoretical consistency guarantees for our light field parametrization.
metrics often keep improving up to and beyond 10 M, the maximum to which we trained the model (especially on more challenging datasets), though qualitative improvements are increasingly subtle as training progresses.

Baseline Details
LFN. We received the LFN [38] code through private communications with the authors. The code was adapted to read the MSN dataset. We trained LFN for 500 k steps on the NMR dataset with batch size 128 and using the Adam optimizer with learning rate 1 × 10 −4 . After training, the weights were frozen, and only the latent codes were optimized for 2 k epochs with batch size 64. For MSN, we trained LFN using a batch size of 16 for 500 k steps as training converged much faster and to lower loss values than with the default batch size of 128. For the camera pose noise ablations, we trained for 350 k steps (those models converged earlier) with batch size 16. After training, we performed latent code optimization for 200 k steps using batch size 16. PixelNeRF. We used the official implementation from the authors [2]. PixelNerf [49] was trained for 1 M steps with the configuration from the codebase used for all experiments in the original paper. During training, each batch contains 4 scenes of 10 images each for the MSN dataset, and 128 rays are sampled per scene for the loss. Training uses Adam with learning rate 1 × 10 −4 . In consultation with the authors, we applied a minor fix in the projection code, tweaking a division operation to avoid NaNs in some experiments.

Appearance Encoding
As the Street View data contains variations in exposure and white balance, we extend our model with an appearance encoder for these experiments. During training, we extract an Reference Render Figure 9. Appearance embeddings -In some scenes, white balance and exposure are not consistent among the input views, see the shift in colors between the reference views in the top row. We add a simple appearance encoder to SRT to handle these cases. Note that the appearance embedding only changes the white balance of the rendered images without changing the structure (bottom row). Further, it generalizes across entirely different views of the scene (e.g., leftmost column). The effect is similar to the appearance handling in [23], however, we do not need an expensive optimization procedure to get the appearance embedding of an image, it is a very fast feed-forward pass, and we do not need hundreds to thousands of images of the same scene to capture appearance variations.
appearance embedding from the respective target views and concatenate it to the input of the final 2-layer MLP after the decoder transformer, see Fig. 2. The appearance encoder consists of 4 blocks of 2×2 mean pooling operations followed by 1x1 convolutions with 32 channels, and a ReLU activation. Finally, we take the mean over the spatial dimensions and map the vector to 4 channels before passing it to the decoder MLP. At test time, the appearance embedding of one of the input images can be used. Fig. 9 shows the effect of the appearance embedding on the output of the model.

Semantic Segmentation
We train the semantics decoder D sem by minimizing the cross-entropy between softmax class predictions and ground truth class distributions p for each ray / pixel: The ground truth classes were generated by running a pretrained 2D semantic segmentation model on the training views, and extracting one-hot predictions.

Benchmark Details
We measure wall-clock performance of all models on a single NVIDIA V100. In Tab Figure 10. Attention visualization for UpSRT -The attention patterns of UpSRT resemble the ones of SRT (Fig. 5). Notably, the model attends into relevant parts of all inputs, showing that UpSRT has learned to use unposed imagery even in complex scenes.
already been onboarded, we measure the rendering FPS in a streaming / interactive setup, i.e. frame by frame, without batching. Finally, C) measures a full application scenario that combines scene encoding and video rendering. Here, we measure the full time taken to onboard a new scene and render a 100-frame video. Note that we allow batching in this scenario, as we assume the camera render path is known, hence the measured times are significantly lower than the sum of scene encoding and single-image rendering times.

UpSRT Model Inspection
UpSRT works without input poses at test time -however, the pose of the first camera is always known by definition, as all target poses are given relative to its reference frame (see Sec. 3). Note that this does not mean UpSRT needs poses -target views must always be defined somehow, and in this case, they are simply defined relative to an arbitrary input view. Hence, the fact that UpSRT works and renders meaningful images by itself does not prove yet that the model is capable of using unposed imagery; the model might only be using the first input camera and ignoring all other input views. In the following, we show that this is not the case, and that UpSRT indeed uses all input cameras. First, we investigate the attention patterns for UpSRT in Fig. 10. As the results show, UpSRT is using all input images for reconstructing novel views of the scene, showing that it is capable of finding similar structures purely based on imagery, and make use of that information for more detailed scene reconstructions. Note that in the encoder, patches in other input images attend strongly specifically into the reference camera, possibly to spread pose information to all other input views. That could also explain why the decoder does not attend strongly into the reference camera.
Second, we compare UpSRT trained and tested on 5 input images with SRT trained and tested only on one input im-age. If UpSRT was only trivially using the first (practically posed) input image, the quality of these two models should be similar. However, we see that SRT only achieves a PSNR / SSIM / LPIPS of 19.2 / 0.53 / 0.52 on MSN in this setup, while UpSRT reaches much better values of 22.2 / 0.65 / 0.41, respectively.
Finally, we take a closer look at the first row in Fig. 4 and notice that the red car is not visible in the first input image (leftmost column), but in other, unposed input views. Despite this, UpSRT is correctly reconstructing the red car at the correct 3D position, showing that the model has detected the position of objects in unposed imagery and has correctly integrated them into the scene representation. Fig. 14 shows further qualitative results for UpSRT.

V-SRT Volumetric Rendering
We show results for V-SRT in Fig. 11. As can be seen, the render quality is comparable to that of SRT (see Fig. 7), with the addition that depth maps can now be acquired as well.
Similarly, Fig. 14 shows results on the MSN dataset. Fig. 13 visualizes the effect of camera noise in our robustness studies, see Sec. 4.4. As can be seen, the amount of noise is subtle, yet enough to severely affect reconstruction quality of geometry-based methods as seen in Fig. 6.

Robustness Study
Temporal Consistency Fig. 8 shows EPI for a dolly motion on an MSN scene. The results show that SRT learns a coherent 3D scene. Note that this consistency also implies that depth maps could be extracted from our model [34,47,45], see also [38].

Comparison to NeRF-based Methods
In contrast to SRT and the baselines we compare to in the paper, NeRF needs costly per-scene optimization, and is known to fail in the few-image setting. In Fig. 12, we compare SRT with Mip-NeRF [3] (improved version of NeRF) and DietNeRF [15] (specialized to handle few images), and see that they both fail for this highly challenging dataset with only 5 images.

Input Views
Nearby V-SRT Predictions Far V-SRT Predictions Figure 11. V-SRT results on Street View -V-SRT has similar quality to SRT (see Fig. 7), but is much slower at inference due to the sampling procedure for volumetric rendering. However, it allows one to render depth maps, which look reasonable in most scenes. In the center example, the depth of the car is inaccurate as a result of the lighting variation, note that e.g. the car changes color due to the metallic surface (e.g., input views 2 vs. 3), and since we did not include the viewing direction (see Sec. 4.3), the model has learned to account for this in the geometry instead.