{'fraction_non_alphanumeric': 0.0339188704740369, 'fraction_numerical': 0.015001516697460221, 'mean_word_length': 4.982184097657539, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes

We present a multi-view inverse rendering method for large-scale real-world indoor scenes that reconstructs global illumination and physically-reasonable SVBRDFs. Unlike previous representations, where the global illumination of large scenes is simplified as multiple environment maps, we propose a compact representation called Texture-based Lighting (TBL). It consists of 3D meshs and HDR textures, and efficiently models direct and infinite-bounce indirect lighting of the entire large scene. Based on TBL, we further propose a hybrid lighting representation with precomputed irradiance, which significantly improves the efficiency and alleviate the rendering noise in the material optimization. To physically disentangle the ambiguity between materials, we propose a three-stage material optimization strategy based on the priors of semantic segmentation and room segmentation. Extensive experiments show that the proposed method outperforms the state-of-the-arts quantitatively and qualitatively, and enables physically-reasonable mixed-reality applications such as material editing, editable novel view synthesis and relighting. The project page is at https://lzleejean.github.io/TexIR.


Introduction
Inverse rendering aims to reconstruct geometry, material and illumination of an object or a scene from images. These properties are essential to downstream applications such as scene editing, editable novel view synthesis and relighting. However, decomposing such properties from the images is extremely ill-posed, because different configurations of such properties often lead to similar appearance. With recent advances in differentiable rendering and implicit neural representation, several approaches have achieved significant success on small-scale object-centric scenes with explicit or implicit priors [7,38,40,49,54,55,58,60,61]. However, inverse rendering of large-scale indoor scenes has not been well solved. The recovered properties are able to produce convincing results for several mixed-reality applications such as material editing, editable novel view synthesis and relighting. Note that we change roughness of all walls, and albedo of all floors. The detailed specular reflectance shows that our method successfully decomposes physically-reasonable SVBRDFs and lighting. Please refer to supplementary videos for more animations.
There are two main challenges for large-scale indoor scenes. 1) Modelling the physically-correct global illumination. There are far more complex lighting effects, e.g., inter-reflection and cast shadows, in large-scale indoor scenes than object-centric scenes. Although the widelyused image-based lighting (IBL) is able to efficiently model direct and indirect illumination, it only represents the lighting of a certain position [13,17,18,48]. The spatial consistency of per-pixel or per-voxel IBL representations [31,34,53,62] is difficult to ensure. Moreover, such incompact representations require large memory. Parameterized lights [16,32] such as point light, area light and directional light are naturally globally-consistent, but modeling the expensive global light transport will be inevitable [1,43,61]. Thus, simple lighting representations applied in previous works are unsuitable in large-scale scenes. 2) Disentangling the ambiguity between materials. Different configurations of materials often lead to similar appearance, and to add insult to injury, there are an abundance of objects with complex and diverse materials in large-scale scenes. In object-centric scenes, dense views distributed on the hemisphere are helpful for alleviating the ambiguity [14,23,38,41,58,59]. However, only sparse views are available in large-scale scenes, which more easily lead to ambiguous predictions [43,55].
In this work, we present an efficient inverse rendering method for large-scale indoor scenes. Aforementioned challenges are tackled individually in the following. 1) We model the infinite-bounce global illumination of the entire scene with a novel compact lighting representation, called TBL. The proposed TBL is able to efficiently represent the infinite-bounce global illumination of any position within the large scene. Such a compact and explicit representation provides more physically-accurate and spatially-varying illumination to guide the material estimation. Directly optimizing materials with TBL leads to expensive computation costs caused by high samples of the monte carlo sampling. Therefore, we precompute the irradiance based on our TBL, which significantly accelerates the expensive computation in the material optimization process. 2) To ameliorate the ambiguity between materials, we introduce a segmentation-based three-stage material optimization strategy. Specifically, we optimize a coarse albedo based on Lambertian-assumption in the first stage. In the second stage, we integrate semantics priors to guide the propagation of physically-correct roughness in regions with same semantics. In the last stage, we fine-tune both albedo and roughness based on the priors of semantic segmentation and room segmentation. By leveraging such priors, physicallyreasonable albedo and roughness are disentangled globally.
To summarize, the main contributions of our method are as follows: 1. A compact lighting representation for large-scale scenes, where the infinite-bounce global illumination of the entire large scene can be handled efficiently.

2.
A segmentation-based material optimization strategy to globally and physically disentangle the ambiguity between albedo and roughness of the entire scene.

3.
A hybrid lighting representation based on the proposed TBL and precomputed irradiance to improve the efficiency in the material optimization process.

Related Work
Image-based lighting. Debevec et al. [13] first introduced the high dynamic range (HDR) light probe as a omnidirectional lighting representation of a certain position, called IBL, which plays a vital role on inverse rendering and lighting estimation tasks in the computer vision. Barron and Malik [3] fitted the spherical-harmonic (SH) illumination, a parameterized representation of IBL, from a single image. Then they extended this single SH illumination into a set of SH illumination [2]. Zhou et al. [62] predicted the per-pixel SH lighting from a single image. Further, the per-pixel spherical-gaussian (SG) lighting [31] and the per-pixel light probe [34] are applied. Wang et al. [53] proposed a per-voxel SG lighting in the 3D indoor scene. The spatial consistency of illumination is unable to ensure with above lighting representations, especially for 3D largescale scenes. Although light probes are able to project consistent light probes in others positions with known geometry, the efficiency in the optimization process is limited [29]. Our proposed TBL not only reserves the advantage of IBL, modelling infinite-bounce global illumination efficiently, but also is naturally globally-consistent.  [57] optimized a more complex configuration via vertexbased lighting, including point lights, line lights, area lights and a environment map, but this method require user inputs to label where is the light source. Our TBL leverages the HDR texture of the entire scene to represent the illumination at any position within this 3D scene without any user input. Absolutely, the TBL is compact and naturally globally-consistent. Implicit lighting. With great advances in implicit representation, researchers began to use implicit neural networks to represent scenes. In particular, NeRF [39] has shown impressive results on scene representation. It leverages a neural density field and a neural radiance field to model a static small-scale scene. Zhang et al. [61] obtained the incident radiance of a certain position from a pre-trained outgoing radiance field [56]. To go a step further, the neural incident radiance field (NIRF), which represents incident radiance from any direction at any position, have shown great potential on inverse rendering [55] and novel view synthesis [51]. However, without constraints for such powerful implicit representation, the ambiguity between material and lighting is difficult to disentangle. Our explicit TBL is capable of eliminating this ambiguity in a interpretable manner.  Figure 2. Overview of our inverse rendering pipeline. Given sparse calibrated HDR images for a large-scale scene, we reconstruct the geometry and HDR textures as our lighting representation. PBR material textures of the scene, including albedo and roughness, are optimized by differentiable rendering (DR). The ambiguity between materials is disentangled by the semantics prior and the room segmentation prior. Gradient flows in Green Background. achieved significant success in many vision tasks with large-scale real-world datasets, such as object detection, semantic segmentation and depth estimation. Unfortunately, collecting the annotation of materials in real-world is difficult at scale. IIW [4] only labels sparse pairwise reflectance comparison for a single image. Therefore, most methods [8, 31, 33-35, 48, 53, 63] are alternative to use synthetic dataset to disambiguate the ambiguous properties. These approaches training on synthetic datasets struggle to eliminate the inevitable domain gap though several datasets have achieved photo-realistic [34,36,44]. Optimizationbased methods [2,3,7,40,43,49,59,61] have shown impressive results on real-world multi-view images. It is known that the prior of different materials is essential to be leveraged in the optimization process to alleviate the ambiguity between materials. Barron and Malik [3] applied a smoothness term for albedo. Multiple variants of filters are used to smooth material in follow-up works [5,31,34]. Zhang et al. [60] leveraged learning priors from real-world BRDFs and imposed a spatial smoothness prior in the latent space. A similar idea has been adopted by a recent work [61]. Schmitt et al. [46] and Luan et al. [38] assumed similar diffuse albedos to have similar specular ones, and applied a non-local bilateral regularizer for specular albedo. Compared to object-centric scenes [7,40,49,[58][59][60][61], the ambiguity between materials is more widely existent in the largescale scene due to sparser observations and more complex objects. Zhang et al. [57] recovered a constant attribute for each floor, wall and ceiling of an empty room. Most similar to ours, Nimier-David et al. [43] and Haefner et al. [19] also assumed similar semantic regions to have similar roughness and specularity. The first difference is we optimize the roughness in a soft manner instead optimizing a constant value [19,43]. The another difference is we leverage the room segmentation prior, which enables us to recover dif-ferent roughness even for similar semantic regions.
In our three-stage material estimation, we explicitly optimize highlight regions to propagate physically-accurate roughness across image regions of same semantics.

Methodology
As shown in Fig. 2, given a set of calibrated HDR images of a large-scale indoor scene, our method aims to accurately recover globally-consistent illumination and Spatially-Varying Bidirectional Reflectance Distribution Functions (SVBRDFs), which can be conveniently integrated into graphics pipelines and downstream applications. To this end, we propose TBL to represent the global illumination of large-scale indoor scenes (Sec. 3.1). In order to improve the optimizing efficiency and quality of re-rendered images in the material estimation stage, we adopt a hybrid lighting representation based on our TBL (Sec. 3.2). To reconstruct physically-reasonable SVBRDFs, we present a segmentation-based three-stage material estimation strategy (Sec. 3.3), which can handle ambiguity of materials in complex large-scale indoor scenes very well.

Texture-based Lighting
We address the issue of how to represent the illumination of a large-scale indoor scene with TBL. The advantages of TBL respectively are the compactness of neural representation, the global illumination of IBL, and the interpretability and spatial consistency of parametric lights.
The proposed TBL, which is a global representation for the entire scene, defines the outgoing radiance for all surface points. We assume that only the diffuse lighting exists in the scene since the diffuse lighting often dominates the scene. Therefore, the outgoing radiance of one point is typically equal to the value of the HDR texture, i.e., the ob-served HDR radiance of corresponding pixels in input HDR images.
We initially reconstruct the mesh model of a entire largescale scene with off-the-shelf classical MVS techniques, e.g., colmap [47]. Finally, we reconstruct the HDR texture based on the input HDR images. Therefore, the global illumination is queried from any direction at any position through the HDR texture, as shown in Fig. 3.

Hybrid Lighting Representation
According to our proposed TBL, the render equation [22] can be rewritten as follow: where H + denotes hemisphere; x denotes a surface point; ω i denotes inverse incident light direction; ω o denotes view direction; n denotes normal; f r denotes the BRDF of point x; Q denotes the HDR lighting of the intersection point between the known geometry G and the ray r(t) = x + tω i .
In practice, we calculate the Monte Carlo numerical integration with importance sampling [24]. To decrease the variance, a large sample number seems to be inevitable, which will significantly increase computation cost and memory cost in the optimization process. Based on the observation that most noise exist in the diffuse component, we precompute irradiance of surface points for the diffuse component. Therefore, the irradiance is able to be queried efficiently instead of expensive online computation as shown in Fig. 3 (right). The Eq.1 can be rewritten as: where L d denotes the diffuse component and L s denotes the specular component. The f r of L s would be modified as f s . The detailed formulation can be found in the supplementary material. We propose two representations to model precomputed irradiance. One is a neural irradiance field (NIrF), a shallow Multi-Layer-Perceptrons (MLP). It takes a surface point p as input and output the irrdiance of p. Another one is a irradiance texture (IrT), similar to the light map widely used in computer graphics. Based on such hybrid lighting representation consists of precomputed irradiance for the diffuse component and source TBL for the specular component, the rendering noise significantly decrease and the materials are optimized efficiently. Therefore, the diffuse component of Eq. 2 can be modeled as Eq. 3.
where f d denotes the diffuse BRDF of point x and Ir denotes the irradiance of point x.
x -ω i x ′ x Figure 3. Visualization of TBL (left) and precomputed irradiance (right). For any surface point x, the incident radiance from direction −ωi can be queried from the HDR texture of the point x ′ , which is the intersection point between the geometry and the ray r(t) = x + tωi. The irradiance can be directly queried from the precomputed irradiance of x via NIrF or IrT.

Segmentation-based Material Estimation
Instead of optimizing neural material [7,49,54,55,[59][60][61], which is hard to model a large-scale scene with extremely complex materials and is mismatched to the traditional graphics engines, we directly optimize explicit material textures of the geometry.
We use the simplified Disney BRDF model [9] with SV albedo and SV roughness as parameters. Optimizing explicit material textures straightforwardly leads to inconsistent and unconverged roughness due to sparse observations [61]. We address this problem by leveraging the priors of semantics and room segmentation. The semantic images are predicted by learning-based models [11] and room segmentation is calculated by the occupancy grid [15]. Our segmentation-based strategy has three phases. Details of each stage are shown in the following. Stage I: albedo initialization. We optimize a coarse albedo based on Lambertian assumption instead of initializing the albedo as a constant, which is widely used in objectcentric scenes [38]. According to estimated illumination in Sec. 3.2, we can directly calculate the diffuse abledo through Eq. 3. However, it recovers over-bright albedo on the highlight regions, which leads to high roughness in the next stage. Therefore, we apply a semantic smoothness constraint to encourage the similar albedo on same semantic segmentation: where L ss denotes the semantic smoothness loss; c denotes one of the classes of semantics; F denotes the feature image to be smooth and we use the image-space diffuse albedo A in this stage; p denotes the pixel of F ; ⊙ is an element-wise product; M seg denotes the mask of semantic segmentation; ϵ denotes a tiny number. The coarse albedo is optimized by: where β ssa denotes the weight of semantic smoothness loss for albedo. Stage II: VHL-based sampling and semantics-based propagation. In multi-view images, only sparse specular cues are observed, which lead to globally inconsistent roughness [61], especially for large-scale scenes. As shown in Baseline of Fig. 9 and NVDIFFREC [40] of Fig. 6, only the roughness of highlight regions is optimized reasonably. Therefore, by leveraging the prior of semantic segmentation, the reasonable roughness of highlight regions would be propagated into the regions with same semantics. We first render images based on input poses with roughness 0.01 to find virtual highlight (VHL) regions for each semantic class. Then, we optimize the roughness on these VHL regions according to the frozen coarse albedo and illumination. Meanwhile, the reasonable roughness can be propagated into the same semantic segmentation through Eq.6: where L sp denotes the semantics-based propagation loss; R denotes the image-space roughness; quantile denotes the q-th quantiles on VHL regions and we set the q as 0.4 for robustness; M vhl denotes the mask of VHL regions for each class. The roughness can be optimized by: Stage III: Segmentation-based fine-tuning. In the last phase, we fine-tune all material textures based on the priors of semantic segmentation and room segmentation. Specifically, we apply a similar smoothness constraint to Eq. 4 and a room smoothness constraint for roughness, which makes the roughness of different rooms smoother in a soft manner. The room smoothness constraint is formulated by Eq. 8: where L rs denotes the room segmentation-based smoothness loss; M room denotes the mask of different room segmentation; c denotes one of the index of each room.
We do not apply any smoothness constraint for albedo. The total loss is defined as: where β ssr denotes the weight of segmentation smoothness loss for roughness and we use the image-space roughness R in L ss in Stage III.

Datasets
Synthetic dataset. We create a synthetic scene with diverse material and light sources with a path tracer [31]. We render 24 views for optimization and 14 views as novel views, and render Ground Truth material images for each view. The details of the scene can be found in the supplementary material.

Baselines and Metrics
To our best knowledge, there are only a few methods that recover the SVBRDFs from multi-view images for largescale scenes. We compare with the following inverse rendering approaches: (1) The state-of-the-art single image learning-based method: PhyIR [34]; (2) The state-of-theart multi-view object-centric neural rendering methods: In-vRender [61], NVDIFFREC [40] and NeILF [55]. Please note that these object-centric approaches are unsuitable to evaluate on large-scale scenes due to simple illumination representation except for NeILF. For fair comparisons, we integrate their material optimization strategies with our hybrid lighting representation, which is designed specifically for large-scale real-world scenes. 1 InvRender [61] NVDIFFREC [40] NeILF * [55] NeILF [55] Ours GT Roughness Novel View Figure 4. Qualitative comparison on synthetic dataset. Our method is able to produce realistic specular reflectance. NeILF * [55] denotes source method with their implicit lighting representation.
PhyIR [34] InvRender [61] NVDIFFREC [40] NeILF [55] Ours RGB Roughness Albedo Figure 5. Qualitative comparison in the 3D view on challenging real dataset. This sample is Scene 1. Our method reconstructs globallyconsistent and physically-reasonable SVBRDFs while other approaches struggle to produce consistent results and disentangle ambiguity of materials. Note that the low roughness (around 0.15 in ours) leads to the strong highlights, which are similar to GT.
We use Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) [52] and Mean Squared Error (MSE) to evaluate the material predictions and the re-rendered images for quantitative comparisons. Moreover, we use Mean Absolute Error (MAE) and SSIM to evaluate the relighting images rendered by different lighting representations.

Comparisons
Evaluation on synthetic dataset. As shown in Tab. 1 and Fig. 4. Our method significantly outperforms the stateof-the-arts in roughness estimation and our roughness is able to produce physically-reasonable specular reflectance. Moverover, NeILF [55] with our hybrid lighting representation successfully disentangles the ambiguity between material and lighting, compared to their implicit representation. Evaluation on real dataset. More importantly, we conduct the experiment on our challenging real dataset containing complex materials and illumination. The quantitative comparison in Tab. 2 shows our approach outperforms previous methods. Although these methods have approximate re-rendering error, only proposed approach disentangles globally-consistent and physically-reasonable materials. We show the qualitative comparison in the 3D view in Fig. 5 and the 2D image views in Fig. 6. PhyIR [34] suffers from poor generalization performance due to a large domain gap and fails in globally-consistent predictions. In-vRender [61], NVDIFFREC [40] and NeILF [55] produce blur predictions with artifacts and struggle to disentangle   Figure 6. Qualitative comparison in the image view on challenging real dataset. From left to right: Scene 8 and Scene 9. Red denotes the Ground Truth image. Our physically-reasonable materials are able to render similar appearance to GT. Note that Invrender [61] and NeILF [55] do not produce correct highlights, and NVDIFFREC [40] fails to distinguish the ambiguity between albedo and roughness.

Ablation studies
To showcase the effectiveness of proposed lighting representation and material optimization strategy, we ablate the TBL, hybrid lighting representation, albedo initialization in Stage I, VHL-based sampling and semanticsbased propagation for roughness estimation in Stage II, and segmentation-based fine-tuning in Stage III. Effectiveness of TBL. We compare the proposed TBL to SH lighting and SG lighting widely used in previous methods [18,31,53,62]. As shown in Fig. 7, our TBL exhibits high fidelity in high-frequency features. Moreover, we evaluate the relighting error of three re-rendering virtual spheres rendered by different lighting representations in Tab. 4. Except for accuracy, the TBL only costs around 20 MB memory to represent illumination while the dense grid-based VSG lighting [53] costs around 1 GB memory and the sparse grid-based SH lighting, Plenoxels [45], costs around 750 MB memory. Therefore, our TBL achieves best performance both in accuracy and compactness. Source TBL NIrF IrT Figure 8. Ablation study of hybrid lighting representation. This sample is Scene 11. Note that we decrease the resolution of input images and samples in Source TBL for keeping almost same time cost. Effectiveness of Hybrid Lighting Representation. We compare the hybrid lighting representation described in Sec. 3.2 to source TBL. As shown in Fig. 8, Without hybrid lighting representation, the albedo leads to noise and converges slowly. With precomputed irradiance, we can use high resolution inputs to recover detailed materials, and significantly accelerate the optimization process. The IrT produces more detailed and artifacts-free albedo, compared to the NIrF. Furthermore, we also compare to implicit lighting in Tab. 1 and Fig. 4. NeILF [55] with their implicit lighting fails in disentangling the ambiguity between materials and lighting, e.g., The lighting effects are incorrectly recovered as materials in Fig. 4 Figure 9. Ablation study of our material optimization strategy in the 3D mesh view on challenging real dataset. This sample is Scene 11. In baseline, we jointly optimize albedo and roughness.
are shown in Tab. 3 and Fig. 9. The roughness of baseline fails to converge and only the highlight regions are updated. Without albedo initialization in Stage I, the albedo in highlight regions is over-bright and leads to incorrect roughness. VHL-based sampling and semantics-based propagation in Stage II is crucial to recover the reasonable roughness of areas where specular highlights are not observed. Segmentation-based fine-tuning in Stage III produces detailed albedo, makes final roughness smoother and prevent the wrong propagation of roughness between different materials.

Applications
Our final output is a triangle mesh with PBR material textures, which is compatible with standard graphic engines and 3D modeling tools. We demonstrate in Fig. 1 that the proposed approach is able to produce convincing results on material editing, editable novel view synthesis and relighting. Moreover, we show several results of editable novel view synthesis in Fig. 10. Note that the viewdependent specular highlights reasonably change as view changes. Please refer to more results in the supplementary material.

Limitations and Future work
There are some limitations of our method. First, we rely on the HDR images to recover the proposed lighting representation for large-scale scene. To lift this limitation, the jointly optimization of lighting and material will be explored. Second, our VHL-based sampling and semanticsbased propagation relays on the scene exists visible light sources. If light sources are not captured, our method leads Nearest View Edited Novel View Scene 9 Scene 8 Figure 10. Editable novel view synthesis. In Scene 8, we edit the albedo of the wall, and edit the roughness of the floor. In Scene 9, we edit the albedo of the floor and the wall. Our method produces convincing results (see the lighting effects in the floor and wall).
to recover bright albedo and low roughness. In such cases, we have to leverage the learning prior to alleviate the ambiguity of materials. Finally, although the geometry reconstructed by MVS is enough for our method, a more accurate geometry would lead to more accurate predictions.

Conclusion
In this paper, we propose a novel inverse rendering framework that recovers globally-consistent lighting and materials from posed sparse-view images and geometry for large-scale scenes. Our texture-based lighting, which not only represents infinite-bounce global illumination but also is compact and globally-consistent, is suitable for modelling illumination of large-scale scenes. Our material optimization strategy leveraging semantics and room segmentation priors is able to reconstruct physically-reasonable and globally-consistent PBR materials. Such a triangle mesh with material textures is compatible with common graphic engines, which benefits several downstream applications such as material editing, editable novel view synthesis and relighting. Table 5. Comparison of costs. N denotes the number of images. Our method achieves competitive performance on costs compared to the highly efficient method, NVDIFFREC [40].

A.1. BRDF Model
In Sec. 3.2 in the main paper, f d and f s are defined as: where A is albedo; l denotes light direction; n denotes normal; v denotes view direction; D denotes Normal Distribution Function (NDF); F denotes Fresnel function and G is the Geometry Factor. We adopt a simplified D, F and G [24,34]. The specular D: The specular F : The specular G:

A.2. Implementation
We use neural networks to predict the depth image [21] and semantic segmentation [11] for each input image. The Figure 11. Overview of our synthetic dataset. It contains diverse materials and objects.
3D mesh of whole scene is reconstructed with depth images and poisson surface reconstruction algorithm [25]. The room segmentation is calculated by occupancy grid [15].
We use 2048 samples to precompute the irradiance of sampled surface points. The NIrF is trained for 2000 epochs with the batch size of 16 and the total size of 1024 and we use the Adam optimizer [26] with a learning rate of 1e-4. The resolution of IrT is 1024×1024.
In material estimation, we use the Adam optimizer [26] with a learning rate of 3e-2 for 40 epochs in all three stages. We set β ssa as 10 in stage 1, set β sp as 1 in stage 2 and set β ssr as 0.1 in stage 3. The resolution of albedo texture to be optimized is 2048×2048 and the resolution of roughness texture to be optimized is 4096×4096. We use 16 samples to re-render the specular component in material estimation. Considering the efficiency of optimization and the natural global illumination of proposed TBL, we apply nvdiffrast [27] with deferred shading to backward the gradient of image-space materials into corresponding textures. We note that nvdiffrast is orthogonal to our pipeline, which can be replaced by other differentiable renderers [20,30,37]. The pre-computed IrT takes around 10 minutes and the optimization process of material takes around 20 minutes.

B.1. Synthetic Dataset
As described in Sec. 4.1 in the main paper, to enable more comprehensive analysis, we create a synthetic scene with diverse material and light sources with a path tracer [31]. As shown in Fig. 11, the virtual scene is consists of three rooms and several objects with different materials. We generate 40 HDR panoramas, and corresponding poses, se-  Table 6. Detailed quantitative comparison on our challenging real dataset. Although NVDIFFREC [40] reaches similar performance to our method, it fails to distinguish the ambiguity between albedo and roughness.

Method
InvRender [61] NVDIFFREC [40] NeILF mantic segmentation, depth, albedo and roughness annotations, and the entire geometry. We use 24 views as input and others as novel views for the novel view synthesis.

B.2. Full-HDR Real Dataset
As described in Sec. 4.1 in the main paper, we capture 10 Full-HDR real indoor scenes due to the lack of Full-HDR real dataset. We first use neural networks to predict the corresponding depth images, and leverage SFM and MVS [47] to reconstruct the 3D mesh with the RGB texture. As shown in Fig. 12, 3D indoor scenes are reconstructed. Note that each indoor scene only contains 10 to 20 images. Therefore, the inverse rendering on these real scenes is extreme challenging.

C.1. Postprocessing
We change the albedo and roughness of ceiling and lamps as a postprocessing on synthetic dataset. We empirically found that the predictions of each approach on these regions are easily prone to local minimal. Based on the observation that the roughness and albedo of ceiling is high in most scenes, we set the roughness of ceiling as 0.8 and the albedo as 0.9. Please note that we update the results for each method on synthetic dataset.  Figure 13. Additional samples of applications. We edit the roughness of floors in Scene 1, Scene 4 and Scene 6, and the albedo for all scenes. Compared to source images, our method still reproduces realistic and consistent lighting effects after editing.
InvRender [61] NVDIFFREC [40] NeILF * [55] NeILF [55] Ours GT Novel View Rendering Roughness Albedo Figure 14. Additional samples of qualitative comparison on synthetic dataset. Our method reconstructs globally-consistent and physically-reasonable SVBRDFs while other approaches struggle to reduce ambiguity of materials.  Figure 15. Ablation study of three-stage material optimization on synthetic dataset.

C.3. Additional Results for Applications
In Sec.4.5 in the main paper, we demonstrate the capability of our method on several mixed-reality applications, such as material editing, editable novel view synthesis and relighting. We show more results on these applications in Fig. 13. Benefiting from our triangle mesh and PBR materials output, which is compatible with standard engines, we can easily edit the properties in a physical manner. We change the albedo or roughness according to the semantic segmentation, e.g., the wooden floors become ceramic floors by changing the albedo of floors. Furthermore, we are able to render physically-reasonable novel views based on our 3D geometry and material textures, which is orthogonal to material editing, as shown in third column in Fig. 13. Last but not least, the entire scene can be rendered under new different illumination, as shown in last column in Fig. 13. Please refer to supplementary videos for more animations.

C.4. Additional Results on Synthetic Dataset
We provide more qualitative comparisons on synthetic dataset in Fig. 14. Our approach is superior than other inverse rendering methods on roughness estimation. And our physically-reasonable and globally-consistent SVBRDFs are able to produce realistic novel views. Note that NeILF [55] with our hybrid lighting representation more successfully disentangles the ambiguity between materials and lighting than NeILF * [55] with their implicit lighting representation.

C.5. Additional Results on Real Dataset
As shown in Tab. 2 in the main paper, our approach outperforms previous neural rendering methods. The detailed results of each real scene are shown in Tab. 6. Note that we do not compare to PhyIR [34] with rerendering error because it uses LDR panoramas as input. Although NVDIFFREC [40] reaches competitive performance to our method, it fails to distinguish the ambiguity between albedo and roughness in Fig. 17, Fig. 18 and Fig. 19. Our approach is able to reconstruct physicallyreasonable and globally-consistent SVBRDF. Such properties re-render similar specular reflectance to GT with less wrong highlights in albedo, which proves we disentangle the ambiguity of materials successfully.

C.6. Additional Results for Ablation studies
We showcase the effectiveness of our three-stage material optimization on synthetic dataset in Fig. 15. As described in Sec. 4.4 in the main paper, the Baseline only update the highlight regions of roughness. Without Stage I, the roughness leads to incorrect result. Without Stage II, the performance of roughness estimation will decrease dra- matically. Without Stage III, the abledo is over-blur and the roughness is unsmooth. As shown in Fig. 7 in the main paper, we show one sample for ablating the effectiveness of hybrid lighting representation. We show more results in Fig. 16. The proposed IrT recovers detailed albedo with less noise.
Additionally, we show more ablation studies of our material optimization strategy on real dataset in Fig. 20 and Fig. 21.

C.7. Bad Cases
As described in Sec. 4.6 in the main paper, our method lead to recover bright albedo and low roughness when the light source is not captured. In Scene 8 in the Fig. 18, we reconstruct over-high albedo and over-low roughness nearby the window because the sun is not captured. The learning prior will be helpful for disentangling the ambiguity between materials in such cases.

D. Broader Impacts
As described in the main paper, our method is able to produce realistic and physically-reasonable images with modified materials or illumination. Therefore, creating deepfake is a major potential negative impact. We can limit the target scenarios to prevent malicious use cases.

Roughness Albedo
Roughness Albedo Roughness Albedo Figure 18. Additional samples of qualitative comparison in the 3D mesh view on challenging real dataset. From top to down: Scene 7, Scene 8, Scene 9 and Scene 11. Our method reconstructs globally-consistent and physically-reasonable SVBRDFs while other approaches struggle to produce inconsistent results and reduce ambiguity of materials.