{'fraction_non_alphanumeric': 0.0348919276576974, 'fraction_numerical': 0.009858844287604763, 'mean_word_length': 4.80550576184379, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion

We present neural radiance fields for rendering and temporal (4D) reconstruction of humans in motion (H-NeRF), as captured by a sparse set of cameras or even from a monocular video. Our approach combines ideas from neural scene representation, novel-view synthesis, and implicit statistical geometric human representations, coupled using novel loss functions. Instead of learning a radiance field with a uniform occupancy prior, we constrain it by a structured implicit human body model, represented using signed distance functions. This allows us to robustly fuse information from sparse views and generalize well beyond the poses or views observed in training. Moreover, we apply geometric constraints to co-learn the structure of the observed subject -- including both body and clothing -- and to regularize the radiance field to geometrically plausible solutions. Extensive experiments on multiple datasets demonstrate the robustness and the accuracy of our approach, its generalization capabilities significantly outside a small training set of poses and views, and statistical extrapolation beyond the observed shape.


Introduction
Enabling free-viewpoint video of a human in motion, based on a sparse set of views, is extremely challenging, but has many applications. Our work is motivated by a breadth of transformative 3D use cases, including immersive visualization of photographs, virtual clothing and try-on, fitness, as well as AR and VR for improved communication or collaboration. However, so far static scenes and rigid objects have been the primary subject of research. In pursuing realistic novel-view synthesis two schools of thought have been established: 1) 3D reconstruction methods aim to recover the geometry of the observed scene as accurately as possible, with novel views generated using classical rendering pipelines [31,43]. 2) Image-based rendering techniques [8,11,41] and very recently neural radiance fields [30] primarily aim at image production quality without explicitly constructing an accurate 3D geometric model. While these techniques explicitly or implicitly reconstruct the scene geometry sometimes, this is however not guaranteed to accurately resemble the true scene geometry. We argue that novel-view rendering and reconstruction are two sides of the same coin, and that reliable viewpoint generalization, especially given relatively few input views, would require good quality for both. To this end, we propose a unified model in order to support both robust reconstruction and photo-realistic rendering. Dynamic scenes, especially those capturing a human in motion, add considerable complexity to the problem: while static scenes can be observed from many views by a camera moving through the scene, any configuration of a dynamic scene is typically observed only from sparse views. Moreover, the scene geometry and its appearance may change considerably over time. To cope with few views, some methods integrate scene knowledge over time by warping observations into a common reference frame [36,38]. At test time, the information is warped back to the desired state and rendered from a novel view. Extrapolating to unseen motion, however, remains challenging. For scenes capturing people, this means that only poses seen during training can be rendered at test time. For some applications, however, rendering the subject over a broad range of motions, or in novel poses, is desirable. To make generalisation over poses and views possible, we rely on additional problem domain knowledge in the form of a human body model, imGHUM [5]. imGHUM is an implicit signed distance function (SDF) conditioned on generative shape and pose codes learned from a large corpus of dynamic human scans. In this work, imGHUM is used as the  Figure 1: Overview of H-NeRF. Given a set of images of a human performer collected from a sparse set of calibrated cameras, we train a geometry-aware neural radiance field by co-learning a deformable signed distance field. First, we estimate the body shape β and track the articulated pose θ, T using the implicit statistical human body model imGHUM (orange). imGHUM encodes all 3D spatial points x across frames with a 4D point descriptor (s, d) referencing a canonical frame. Using the foreground mask M, we co-train a residual SDF and a NeRF network in that canonical space, in order to integrate all image observations into a consistent implicit 3D geometry ∆d and its view-depended (v) radiance representation (c, σ). The trainable residual SDF and NeRF networks (in blue) are conditioned on the body pose θ and the root transformation T to model pose-dependent geometric and appearance variations. Our framework supports both accurate 3D geometric reconstruction and free-viewpoint rendering, and generalizes well to novel views, shapes, and poses.
common reference frame for a neural radiance field and as a structured prior for robust reconstruction. Additionally, since imGHUM can represent a broad distribution of statistically valid human poses and shapes, we can render the reconstructed subject in novel poses and even with modified body shapes. In summary, our system supports photo-realistic free-view point temporal rendering of a human subject given only sparse camera observations. By conditioning on an implicit human body model, we can render considerably different viewpoints, body poses, and body shapes compared to those observed in training. Our carefully designed losses ensure not only good image quality but also plausible temporal reconstructions that can be used for the free-viewpoint visualisation of human performance capture.

Related Work
Human performance capture is the process of reconstructing the dynamic (4D) geometry of a human subject as observed in one or multiple synchronized views. Pioneering methods deform a rigged template mesh against silhouettes observed in multiple views [7,51], estimated skeleton key-points [13], or image correspondences [10]. Later systems relying on RGB-D video streams [6] or monocular capture [4,3,18,53,17,16] have also been presented. All these methods rely either on a pre-scanned template mesh or on a body model that is deformed to explain the image evidence.
Neural representation for view synthesis. With the advent of neural networks, researchers have begun to explore alternative solutions to represent a scene in order to support novel view synthesis and photo-realistic rendering; we refer the reader to [49] for a survey. Different scene representation have been explored. Some methods use voxel grids to embed the scene [24,46]. For rendering, the voxel grid is probed by shooting a ray and by linearly interpolating voxel values. These samples are then transformed into color values using a neural network. Others have proposed neural textures [50,44] that can be rendered based on view-dependent effects or texture synthesis networks [23] to realistically render meshes. In a similar spirit, other methods render point clouds, where each point carries local appearance information [27,2]. Riegler and Koltun [41] reproject features from nearby views and rely on a SfM reconstruction scaffold for novel view synthesis. With the recent advent of implicit function networks [26,35,9,28], such 3D representations have been explored for rendering and view synthesis with great success. In contrast to discrete voxel representations, textures, or point clouds, these methods represent the scene as a continuous function and thus are not bound to a specific image or volume resolution. In the pioneering work of Sitzmann et al. an implicit function produces features displayed using a neural renderer [47]. Follow up methods focus on 3D geometric reconstruction [45], and use 2D supervision [32,54].
Neural Radiance Fields (NeRFs) are a recent approach to represent scenes for novel view synthesis. Mildenhall et al. [30] introduce Neural Radiance Fields resented as fully connected neural networks, where the input is a spatial query point and a viewing direction. The output is a volume density and the emitted radiance at the query location in the direction of the viewer. By ray-tracing using this simple representation, one can generate photo-realistic images from novel views. Despite the excellent quality of results, one drawback is the slow rendering time. To this end, researchers have presented faster versions that e.g. transform the radiance field into more efficient sparse grids [19] or remove the dependence on viewing-direction during rendering by estimating a spherical harmonic representation of the radiance function [55]. Others improve fidelity or rendering time by tackling ambiguities in the original formulation [58], spatial decomposition into multiple NeRFs [40], or combining NeRF with sparse voxel fields [22]. Initial work on adapting NeRF to dynamic scenes has been presented as well. Park et al. [36] produce "Nerfies" (NeRF-Selfies) from videos where subjects carefully move a camera around their head. The scene information is fused by warping query points into a canonical reference frame. Similarly, Pumarola et al. [38] produce dynamic NeRFs from synthetic animation data. Related to our approach, some methods integrate human body models to fuse information over time. A-NeRF [48] uses a skeleton to rigidly transform NeRF features to refine estimated 3D poses. A similar approach is followed in NARF [33] for view synthesis. Most related, Neural Body [37] attaches learnable features to the vertices of a SMPL body model [25]. These features are processed using a sparse 3D convolutional network, where the output forms a neural radiance field. In contrast to our approach, the resolution is bounded by the spatial resolution of the 3D convolutional network and no geometric supervision is used. We highlight differences in §5.

Background
Given a collection of images capturing a dynamic scene of a human in motion, observed from a sparse set of calibrated camera views (in the limit a monocular camera, as we will show), we aim to learn both the detailed temporal geometry of the human in motion, and to render the sequence from novel camera views and for different human poses. To this end, our work unifies two main methodologies: 1) implicit 3D human representations, and 2) volumetric radiance fields. In this section, we provide the relevant background on both representations, which we co-learn in a joint framework.
Neural Radiance Fields. A neural radiance field (NeRF) [30] represents a 3D scene as a continuous function of color volume densities. The model consists of a neural network function F ω that maps a 3D spatial point x ∈ R 3 and a viewing direction v ∈ R 3 to a volume density σ ∈ R + and a radiance c(x, v) ∈ R 3 emitted towards the viewer. In practice, NeRF encodes the inputs x and v using a sinusoidal positional encoding γ : R 3 → R 3+6m that projects a coordinate vector into a high-dimensional space using a set of sine and cosine functions of m increasing frequencies. Given a ray r = o + sv with N samples {x} originating from a camera location o, NeRF integrates radiance values along the ray by means of alpha blending. The pixel/ray color is approximated with numerical quadrature [34], where α(x i ) is the transparency by accumulating transmittance along the ray, and δ i = |x i+1 − x i | is the distance between adjacent samples. The NeRF function F w is fully differentiable and its network parameters w can be optimized using an image reconstruction loss [30]. An approximate 3D scene geometry S F = {x|σ(x) = σ h } can be extracted from the trained opacity field via Marching Cubes [21] at a density threshold σ h .
Implicit Generative Human Models. Implicit human body surfaces are typically represented as the decision boundary of either binary occupancy classifiers [12,42,29] or signed distance functions [15,5]. Specifically, our work builds upon the SOTA statistical implicit human model imGHUM [5] H ω : (T −1 x, β, θ) → (d, s) that maps a 3D spatial point x, unposed with root joint transformation T ∈ R 4×3 , to its signed distance value d ∈ R with respect to a body surface parameterized with body shape β ∈ R 16 and articulated pose θ ∈ R 118 . In addition to d, imGHUM returns implicit continuous semantics s ∈ R 3 of the query point which corresponds to the 3D coordinate of the nearest surface point defined on a canonical surface. We refer to the original paper [5] for details. Essentially, imGHUM builds a human-centric 4D semantic descriptor (d, s) for all spatial points in the neighborhood of the surface. imGHUM is trained with a large collection of human scans of diverse body shapes and poses, sharing the same generative shape and pose latent code with the mesh-based statistical human model GHUM [52]. The implicit articulated 3D human body S H (β, θ, T) = {x|d(x) = 0} is defined by the the zero-isosurface of the signed distance field.

Method
We present the details of our main contribution, H-NeRF, a novel neural network ( fig. 1) that exploits the power of volumetric radiance fields to learn complex human structure and appearance, by relying on statistical implicit human pose and shape signed distance functions for accurate geometric reconstruction. Further, H-NeRF relies on imGHUM, an implicit model of articulated human pose and shape, as a rich geometric prior, to integrate scene information over time, and to represent human articulation. Co-learning both a radiance field and a signed distance function of scene geometry consistently in a unified framework, enables accurate 3D geometric reconstruction and volume rendering of a dynamic human in motion, through 1) consistent integration of image observations over time, and 2) good generalization capability for novel viewpoints, human poses, and for statistically extrapolated shapes, given only very few training poses and camera views.
Given a video of a human in motion, observed by a sparse set of calibrated cameras, our objective is to generate free-viewpoint video of the observed person and to reconstruct the underlying 4D geometry. The set of input images, with a resolution of w × h pixels, is denoted as {I c t ∈ R w×h×3 |c = 1, . . . , N c , t = 1, . . . , N t }, where c is the camera index, N c is the number of cameras, t is the frame index, and N t is the number of frames. For each image, we apply [14] to obtain the binary foreground human mask M c t ∈ R w×h . In addition, we obtain a temporally consistent imGHUM latent shape and pose codes, β and (θ t , T t ), respectively, at each frame index t, by optimizing an imGHUMequivalent parametric replica under multi-view keypoint and body segmentation losses [57]. We refer to our Sup. Mat. for details on the imGHUM fitting process.
In the sequel we first explain our adaptations to NeRF for the static case. imGHUM is used as a prior in order to bias the reconstruction of the observed scene towards a more accurate human geometry. We continue by explaining the additional methodological innovation needed in the dynamic case. Hereby, imGHUM provides spatial-temporal correspondences and is used as the common reference frame to fuse information across different views and time instances.

Static Semantic Human NeRF
We first formulate H-NeRF for a human capture at a single moment in time (N t = 1). From multi-view image observations, we co-learn a radiance field F ω : x, v → (c, σ) for free-viewpoint rendering, and a signed distance functionĤ ω : x →d for 3D geometric reconstruction. We useĤ ω for the dressed subject in order to distinguish it from the body's imGHUM SDF H ω .
Like NeRF, we formulate an L 1 image reconstruction loss to optimize F w as where R denotes the batched set of all pixels/rays, andC(r), C(r) are observed and rendered pixel colors, cf. (1), respectively. However, under sparse training camera views, the volumetric radiance field is not well regularized, leading to poor generalization to novel viewpoints, cf. fig. 2. Specifically, we observe that the model encounters difficulties in correctly representing the scene and in separating the person from the background. The model fails to learn a semantically meaningful opacity field, i.e. α = 0 in free space, and 1 if occupied.
Coarse Scene Structuring. Using imGHUM fits for all training images, we can spatially locate the person in 3D. We define a 3D bounding box B ∈ [S H − , S H + ] around the detected person, where S H , S H are the minimal and maximal coordinates of the human body surface S and is a spatial margin reserved for geometry not modeled by imGHUM. All radiance points associated to the rendering of the person should reside inside the bounding box, leading to a 3D segmentation loss where b(x i ) is 0 if outside of B and 1 otherwise, and M(r) the image mask. We rely on the mask, so the loss is only applied to image observations from the subject, and not the remaining scene geometry.
Unifying SDF with NeRF. After structuring the scene coarsely, we now couple the estimated radiance field with an implicit signed distance-based 3D reconstruction, in order to further regularize the opacity distribution. A signed distance function associated to the detected person in the image naturally comes with a 3D classifier for all spatial points, whered(x i ) > 0 means x i is in free space, whereas x i lies within the subject whend(x i ) <= 0. We rely on this insight in order to co-learn a SDF of the performer and constrain the radiance field. To this end, we introduce a pseudo alpha valueα(x i ) = φ(γd(x i )) where φ is a Sigmoid activation function and γ controls the sharpness of the boundary. To refine the NeRF opacity semantics, especially for the volume in the neighborhood of the human surface B, we formulate two losseŝ whereĈ(r) is the rendered pixel color with blended alpha valuesα replacing NeRF alpha values α with SDF-based pseudo alphaα for all ray points inside the bounding box B. The first term in L blend requires the rendered pixel color for all the intersecting rays within the human mask to come from the surface, whereas the second term assumes that all background color is formed from ray samples outside of B. We set η = 1 when no other geometry than the person is inside B and tune η down if the assumption is violated (e.g. person standing on a floor). The term L geom uses the binary cross entropy loss to couple the NeRF surface boundary with the zero-isosurface of the signed distance function describing the subject. While the coupling terms given by (7) and (8) act as strong priors for the opacity distribution, during test time, we still rely on volumetric radiance rendering with learned NeRF alpha values α to support transparency effects and complex geometry for structures like hair.
Image-based SDF learning. Learning the signed distance fieldĤ ω from scratch using a sparse set of training images is still challenging. The model often fails to reconstruct reasonable human geometry, even when using coupling losses (7) and (8). We therefore leverage imGHUM as an inner layer for the target human reconstruction and combine it with a light-weight residual SDF network ∆H ω : x → ∆d. The residual SDF models surface details, including hair and clothing, that are not represented by imGHUM. The final signed distance for Given the training images, we learn the personalized residual SDF using our coupling losses, given by (7) and (8), and additionally apply geometric regularization whered min (r) denotes the minimal signed distance for all sampled ray points and ψ is the ReLU activation function. The term L seg ensures that if a pixel is inside the human segmentation mask, there should be at least one intersection between the ray and the 3D human surface and therefored min (r) should be non-positive. Otherwise, all ray samples should have positive signed distances. Using L eik , we enforce the composite SDFd to be approximately a signed distance function, i.e. incorporating Eikonal regularization [15]. The term L reg regularizes the residual distance ∆d to be non-positive. This is because imGHUM should reside within the geometry and personalized geometric details given by ∆d should be modeled on top of the skin surface.

Dynamic Semantic Human NeRF
We now extend the framework to model dynamic human motion. To implicitly model scenes where the human moves, we learn a consistent, continuous function G ω : (c, α,d|z) → (c , α ,d |z ) for both the geometry and the appearance variations across frames. For generalization to novel poses or shapes, G ω should be conditioned on a semantically meaningful latent code z that can be interpolated and should ideally have good extrapolation properties. To integrate scene information over time in a single radiance field, we follow the approaches in [36,38] and warp observations into a canonical reference frame. Given our constraints to G ω , imGHUM, conditioned on its semantic shape and pose latent code (β, θ, T), naturally provides spatial correspondences across frames. Given two spatial points x and x in the space of two human instances (β, θ, T) and (β , θ , T ) respectively, we decide these are in correspondence if they have the same semantics and signed distances (s, d|β, θ, T) = (s , d |β , θ , T ). Essentially, imGHUM assigns a 4D point descriptor (s, d) ∈ R 4 to any spatial point and deforms the volume continuously with respect to the parameterized articulated human body surface. Specifically, we apply H ω : to map a spatial point x to a canonical point descriptor (s, d). We then modify the input to the NeRF function with the point descriptor as F w : (s, d) → (c, σ), and similarly for the residual SDF function as ∆H ω : (s, d) → ∆d. imGHUM is pre-trained using a large corpus of 3D human scans, thus we keep it fixed in our network and directly use this prior to integrate structured geometric and appearance information across training frames. Given that the background scene is invariant to the human motion, we only apply the imGHUM warping function to spatial points inside the bounding box B, thus largely improving computation time and lowering memory usage.
Pose-Dependent Geometry and Appearance. Training a single canonical NeRF and a residual SDF from a sequence integrates image observations into a consistent implicit geometric and volumetric representation of the radiance. However, we observe that fine-grained geometric and appearance variations caused by human motion are missing in the canonical NeRF and the residual SDF. To model such variations in geometry (clothing deformation) and appearance (lighting, self-shadows), we condition the NeRF function F ω -the volume density σ (geometry) and color value c (appearance) -as well as the residual SDF ∆H ω (geometry), on the body pose code θ. In addition, we also notice that the relative position and rotation of the person with respect to the scene largely affects appearance (due to illumination effects), but should not affect the geometry. Based on this observation, we additionally condition the NeRF color on the person's root transformation as c(x|θ, T).
H-NeRF Articulation. Differently from prior work [36,38], our H-NeRF modules are conditioned on semantic human latent codes (β, θ, T), leading to an SDF and radiance field that can be interpolated. Moreover, due to imGHUM's capability to accurately model geometric volume deformation for diverse human poses and shapes, H-NeRF enjoys strong generalisation to novel poses and even body shapes. Concretely, during inference, one can simply assign a different set of (β, θ) to transfer the learned geometry and appearance to an unseen pose and shape configuration. For better generalization, we add Gaussian noise to the input code (θ, T), preventing network overfitting to pose-dependent deformation and appearance effects experienced during training. When only a sparse set of camera views is available, We apply the same technique to NeRF, conditioned on viewing direction v.
Fine-tuning imGHUM Pose and Shape. We observe that imGHUM fitting can be a source of error when learning F ω and ∆H ω from dynamic image sequences. Instead of keeping parameters fixed, we further improve the fit during training by fine tuning a time-consistent shape correction ∆β and per-frame pose correction ∆θ(t) with L fit = r∈R BCE(M(r), d min (r|β + ∆β, θ + ∆θ(t))), Linc = ∆β 2 + 1 Nt where L fit aligns the imGHUM fit with the human foreground segmentation and L inc regularizes the incremental corrections.

Experiments
We quantitatively and qualitatively evaluate H-NeRF through ablations and by comparing with other methods. Please refer to our Sup. Mat. for additional results and ablation experiments. We first detail our model architecture, as well as the datasets and evaluation metrics used.
Architecture and Training. We use the same network architecture for all of our experiments, where the trainable modules F ω and ∆H ω consist of eight 256-dimensional and six 128-dimensional MLPs respectively, with a skip connection to the middle layer and with Swish activation [39]. As in the original NeRF [30], we use 256 coarse and fine-level ray samples, for each of which we use 8and 1-dimensional positional encoding for F ω and ∆H ω , respectively. We train the network using the Adam optimizer with a learning rate of 0.001 exponentially decayed by a factor 0.1 until the   [20] (5 sequences). The first two are synthetic, rendering animated (RenderPeople) or 4D human scans (GHS3D) from four orthogonal cameras, where ground-truth geometry paired with images is available. We evaluate both the image and the 3D geometry reconstruction quality from two novel cameras, and qualitatively demonstrate generalization to shapes and poses not in the training set. The remaining datasets are real captured videos (PeopleSnapshot: monocular, Human3.6M: four cameras) without paired 3D geometry, where we only evaluate the rendered images. For image metrics, we adopt peak signal-to-noise ratio (PSNR ↑), structural similarity index (SSIM ↑) and learned perceptual image patch similarity (LPIPS ↓) [59]. For evaluation, we render the NeRF field inside the human bounding box B and compare to the ground-truth segmented image within a region of interest around the person. To evaluate geometric reconstruction quality, we report bi-directional Chamfer (Ch) L 2 distance, Normal Consistency (NC) and Volumetric Intersection over Union (IoU), evaluated on the mesh produced by applying Marching Cubes on the SDF, with a resolution of 256 3 .

Static Human Reconstruction under Sparse Viewpoints.
H-NeRF learns a geometrically regularized volumetric radiance field constrained by an imGHUM-based SDF, which significantly improves robustness under sparse training camera views. In fig. 2, we show reconstruction of a static Render-People and GHS3D scan, respectively, using the original NeRF [30], the state-of-the-art multi-view reconstruction approach IDR [54], and H-NeRF, for increasing number of cameras. Both novel view image quality and geometric accuracy improve for all methods as the number of training views increases. However, in contrast to competitors, H-NeRF produces good quality output even under sparse training views (2)(3)(4)(5)(6)(7)(8), demonstrating the generalization capability of a co-training approach.
Dynamic Human Reconstruction and Rendering. In the following, we evaluate H-NeRF's capabilities to reconstruct and render dynamic scenes ( fig. 5). We compare H-NeRF against NeuralBody [37] (tab. 1), the current state-of-the-art for novel view synthesis of humans in motion. For fair comparisons, we have trained NeuralBody with the same data as H-NeRF and use GHUM meshes (instead  of SMPL). GHUM meshes with the same pose and shape representation as in our model are used for NeuralBody's structured latent codes. We further compare with Nerfies [36]. However, Nerfies has difficulties with the high range of motion in our test sequences and fails for most of them. We consider such results less meaningful and we report them only in the Sup. Mat. for completeness.
H-NeRF overperforms NeuralBody across datasets and metrics, especially for geometric reconstruction. One possible explanation for NeuralBody's lower performance on view synthesis, is that NeuralBody is not conditioned on the root transformation. Thus, global lighting effects are handled less well. Furthermore, pose depended effects that go beyond body articulation are only indirectly modeled through relative distances of input mesh vertices. Finally, NeuralBody conditions on the frame index, which is set to zero for novel poses. On the PeopleSnapshot dataset where no view-dependent and only subtle pose-dependent effects are present, these limitations are not entirely apparent. In contrast, H-NeRF pays special attention to modeling both detailed geometry and appearance, and its explicit conditioning on pose and the root transformation captures pose-dependent geometric and appearance effects. This strategy pays off, especially in more complex scenarios.

Frame Number Ablation.
We have illustrated H-NeRF's rendering and reconstruction capabilities for static and dynamic sequences. Next, we evaluate the necessary number of different poses or time-frames H-NeRF needs during training for good pose generalisation. We again compare against NeuralBody. To this end, we have trained both methods with an increasing number of images, uniformly distributed over the full sequence. Results are shown in fig. 3. As expected, both methods perform better when trained with more data. However, H-NeRF performs overall better and more importantly, the quality degrades less in the sparse training regime, especially for geometric reconstruction. Our results support H-NeRF's robustness to sparse training data and its capability to reconstruct accurate geometry. In practical terms, H-NeRF can be robustly trained with as little as 10 temporal frames per camera (in a four camera set-up), resulting in a performance capture effort of only a few minutes.
Qualitative Results for Pose Generalisation and Shape Extrapolation. Finally, we illustrate H-NeRF's rendering and reconstruction accuracy qualitatively ( fig. 4). Consistently with the reported metrics, our synthesized novel poses appear sharper, more detailed, and contain less noise than the current state-of-the-art. The estimated geometry is complete, smooth, and contains much of the detail present in the original scan, e.g. the clothing folds on the back of the person on the left, the first row.
In contrast, geometries produced by NeuralBody are noisier and sometimes incomplete. To further Limitations and Broader Impact. In the current setup, we assume full-body views of a single person in every-day clothing. Inconsistency with these assumptions, e.g. by placing another person in the scene and occluding the performer, would break the method although models of partial views, or representation estimates of multiple people could be used for generality. Furthermore, our method exhibits some sensitivity to the estimated body shapes and poses, as well as the quality of image segmentation. Although the process of learning to render could absorb certain inaccuracies, as pose, shape or segmentation are increasingly degraded, the method would fail eventually.
Our method paves the way towards immersive AR and VR applications. In contrast, our approach is not targeted, or particularly useful for applications like visual surveillance or person identification as a particular set-up and a cooperating subject is needed. We do not build an audio-visual model that would be necessary for deep fakes.

Conclusions
We have presented novel neural radiance field models (H-NeRF) for the photo-realistic rendering and the temporal reconstruction of humans in motion. Our objective is to extend the scope of the previous state of the art, focused on static scenes observed by a large number of cameras, by building dynamic models, which based on a sparse set of views, can generalize well to novel camera views, human body poses, and extrapolate over human body shapes. Our key contribution is in developing a new model with associated multiple losses, in order to specialize and constrain a generic NeRF formulation by using a compatible implicit statistical 3D human pose and shape model, represented using signed distance functions. Our implicit geometric formulation captures not just the statistical regularities of the human body, but also hair and clothing represented as an implicit residual network. Our model is trained end-to-end based on several novel losses, and achieves good results for both 3D reconstruction and photorealistic rendering. Training the body model in an end-to-end rendering framework carries the promise to learn complex implicit skinning functions based on images only, a performance previously possible only in the realm of human capture using complex and expensive 3D body scanners, in the laboratory. We illustrate the favorable capabilities of H-NeRF through extensive experimentation using several datasets and against other state of the art techniques.

Supplementary Material
In this supplementary material we provide additional results, both quantitative and qualitative, as well as comparisons with other methods, ablation studies, and additional background on our model components.

A Additional Results
Dynamic Human Reconstruction and Rendering. In fig. 6, we provide qualitative results for samples taken from the four datasets we use for experiments. Our method shows very good accuracy for both novel view synthesis and for 3D geometric reconstruction. In addition, our model can generate novel geometry corresponding to poses not in the training set, by altering the input imGHUM pose control (θ, T), as shown in the fig. 6. We note that differently from novel view synthesis for a fixed pose, the source of error in the image synthesis of a novel pose (not in the training set) could arise from both the NeRF rendering function and the implicit geometric surface skinning. For our RenderPeople sequences where ground-truth images (renderings) and geometries for the novel test poses are available, we quantitatively report both image and geometric errors in tab. 1 of the main paper. We also show, side-by-side, qualitative comparisons in fig. 4 (main paper) and fig. 6 of this material.
Tab. 2 provides additional information on the used datasets. Specifically, the RenderPeople sequences are generated by animating a single rigged scan using motion capture from the CMU [1] and Hu-man3.6M [20] datasets, respectively. The GHS3D dataset is composed of 14 sequences dynamically capturing dressed subjects undertaking freestyle motions (e.g. presentation, dancing or exercising). The PeopleSnapshot [4] dataset consists of monocular videos of a subject rotating in front of a static camera. We select one rotation cycle for training and another (different) cycle for testing. We also demonstrate the capabilities of our method on the Human3.6M dataset which captures the scene using four synchronized and calibrated cameras. There we use the first 320 frames for training (sampled every 8th frame) and the following 160 frames for testing (sampled every 4th frame).
Comparison to Nerfies [36]. For large full-body articulated human motions, as in our typical use case (and illustrated by our sequences), Nerfie overfits to the training images by mixing the human geometry with the background. This results in extremely limited generalisation for both novel view synthesis and 3D geometric reconstruction (tab. 3). We notice that geometry is effectively not reconstructed, leading to zero IoU. Moreover, similarly to D-NeRF [38], the deformation function is conditioned on time (or an embedding of the video frame index), hence the method cannot generalize to novel poses or shapes. To evaluate image (rendering) quality, we crop the image using a 2D bounding box computed from the ground-truth image segmentation (padded with a 20 pixel border), as our region of interest. For H-NeRF, IDR [54] and NeuralBody [37], the evaluation is performed by comparing to the cropped ground-truth image and using the foreground human segmentation. However, for the original NeRF and Nerfies, segmentation is not produced and therefore we compute metrics on the image with background.
Generalization. In addition to pose generalization shown in fig. 6, we further evaluate generalization to novel shapes in fig. 7 for sample sequences taken from the four datasets. The shape extrapolation is achieved by replacing the training imGHUM shape latent code with a different β, and propagate the shape changes to both image rendering and the geometry. To understand this process, one can consider imGHUM as the inner layer of the human body. The color and signed distance of a spatial point is deforming w.r.t. the body surface. Therefore when we change the underlying body shape, the NeRF radiance field and the signed distance function are updated accordingly. We observe high-quality visual shape generalization results produced by H-NeRF, in fig. 7, even for significant volume changes w.r.t. the ground truth shape.
In fig. 8, we study the image synthesis quality as a function of the differences between testing and training camera views. The view difference (x-axis) is evaluated by computing the angle between the two vectors from the camera position to the 3D center of the person. All methods (H-NeRF, IDR and NeRF) show a degradation of quality as viewpoints deviate significantly from training, but the view synthesis capability of H-NeRF is consistently better than the other two, likely given its capacity to estimate a reasonable 3D surface geometry. The increased error is largely due to the rendering of body parts not visible in the training images. We note that for our RenderPeople and GHS3D (static   gt-image and gt-geom indicate that ground-truth image and geometry are available, respectively. If opt is yes, we perform fine-tuning of imGHUM during training. Figure 7: Qualitative evaluation of shape extrapolation for the four datasets. The leftmost is the training shape and we extrapolate both the rendering and the geometric reconstruction to four novel shapes (three for PeopleSnapshot). The shape volume change w.r.t. the learned shape for the 4 rows are: −5%, +29%, +50%, +87% ; −27%, +48%, +18%, +32%; −41%, +18%, +9%; −15%, +4%, +18%, +44%.
or dynamic) sequences, the four training cameras are on the four sides of the subject, residing on a sphere with radius 2.4m, oriented towards the center as shown in the last row of fig. 8, where we also show two typical test camera views for our dynamic sequences.
We further evaluate the pose generalization capability of H-NeRF as a function of the pose difference to the training configurations. Fig. 9 shows the image and geometric metrics of two RenderPeople sequences for all of our novel testing poses. We do not observe strong correlation as a function of pose differences w.r.t. training. and suspect that both image and geometric quality are largely affected by the accuracy of imGHUM on test poses, and less so by pose differences themselves. However, one still needs to provide sufficient examples for learning pose-dependent geometry and appearance as shown in our frame number ablation experiment ( fig.3 in the main paper).
Monocular videos. The PeopleSnapshot dataset consists of monocular videos of people rotating to a full circle in front of the camera. However, the subjects are consistently holding an A-Pose. We therefore additionally evaluate H-NeRF on two RenderPeople and two GHS3D dynamic sequences with large pose variations but trained with only a single camera. Tab. 4 and fig. 10, respectively, show numerical and visual results. Our method still shows strong robustness for novel view synthesis, geometric reconstruction and even pose extrapolation, although at some quality expense compared to models trained using four cameras. We notice that GHS3D-S47 shows a more significant drop in its metrics when trained in the monocular case, which may be caused by the back of the subject not being fully visible in the training images (whereas for the other sequences, the subject is turning in front of the camera). This experiment clearly demonstrates the capability of H-NeRF to integrate temporal image information into consistent neural radiance and SDF representations.
Similarly to NeuralBody [37], we rely on good pose estimation for consistent integration of image observations across multiple dynamic frames. Because of degraded monocular pose estimation, we excluded a small number of side view frames from the PeopleSnapshot dataset aiming to maximize the model quality. Note that for fair comparisons, we trained NeuralBody with the same images and poses as our method. In addition, we have trained a RenderPeople sequence with all frames and have indeed observed degraded novel image synthesis quality caused by the pose estimation inaccuracies (PSNR ↑: 27.5 (full) vs. 28.1 (filtered), SSIM ↑: 0.81 vs. 0.86, LPIPS ↓: 0.23 vs. 0.21). Our process of fine-tuning imGHUM pose and shape (Eq. 11 in the main paper) helps alleviate the problem (see Tab 5) and any other better pose estimation technique would be complementary to our approach.

B Ablation Studies
Tab. 5 shows ablation studies for each loss for our four datasets. Training H-NeRF with all proposed losses strikes a good balance between image rendering quality and geometric reconstruction accuracy, and achieve the best performance for most metrics. For example, we observe significant drops in image quality when we do not condition the NeRF appearance under the root transformation. We largely benefit from fine-tuning the imGHUM parameters for the PeopleSnapshot dataset where we have the highest geometric fitting error given the use of only monocular videos. The terms L blend and L mask impact the image metrics significantly. For example, without L blend , the geometric metric is slightly better but leads to significantly worse SSIM for the real-world videos. The terms L geom , L reg and L eik impact the geometric metrics more, and we observe considerably worse reconstruction performance when turning these off. The Eikonal regularization helps smoothing the surface reconstruction and is critical for videos with insufficient views (PeopleSnapshot) or lower image resolution e.g. due to people placed farther away from the camera (Human3.6M). The term L seg affects both rendering and geometry and we clearly see performance drops in its absence.

C Training, Memory Consumption and Timings
We have trained the models for 10k iterations of 4k ray batch size on 8 Nvidia v100 GPUs, which takes about 6-8 hours for each dynamic sequence. For an image of 512 × 512 resolution, the inference for H-NeRF on a single Nvidia v100 GPU takes about 9.1 sec. The main computation overhead compared to the original NeRF formulation (about 6.5 sec) comes from the imGHUM warping. imGHUM warping also limits the maximum number of query points to be 64 3 due to memory constraints (i.e. 1024 rays consisting of 128 coarse and 128 fine samples). We utilize the coarse scene structuring such that we only query imGHUM for points inside the 3D bounding box B. For exterior points (static background or free space) we use the original position coordinate and a constant distance value (1.0) as input features to the NeRF network. In practice, this significantly reduces the number of imGHUM query points by 90% and largely alleviates the memory constraints.

D imGHUM Fitting
imGHUM shares the same shape and pose latent code with the mesh-based generative human model GHUM [52]. To fit the imGHUM parameters to a given image, we therefore use an off-the-shelf fitting approach for GHUM [56,57]. We subsequently use the GHUM fit (pose and shape) in order to initialize imGHUM, which shares the same latent code. Specifically, given a monocular image or a set of images collected from multiple views, we use the neural network HUND [57] that takes a cropped human detection, and outputs 137 keypoints with confidences and 15 body-part semantic segmentation masks. HUND further predicts the GHUM shape β and pose values θ based on the various semantic features (keypoints and segmentation masks) extracted from the image. Given HUND predictions, we follow up with a kinematic optimization that better aligns GHUM's predicted keypoints and semantics segmentation with corresponding primitives extracted in the calibrated input cameras. The optimization is formulated as image alignment for which we adopt the self-supervised keypoint and body segmentation loss from [56] (cf. their eq. 11), with L 2 regularization on the latent embeddings of the body shape β and pose θ. The temporal smoothness is applied with a L 2 loss on the pose differences between neighboring frames.          Table 5: Ablation study on losses (Chamfer ×10 −3 ). The last three columns (-opt, -T, -noise) show results without imGHUM fine tuning, not conditioning the NeRF appearance using a root transformation T, and not applying Gaussian noise to NeRF's condition code (θ, T, v), respectively. We do not apply imGHUM parameter fine-tuning on the RenderPeople sequence.