{'fraction_non_alphanumeric': 0.028458391404037227, 'fraction_numerical': 0.012992851683675763, 'mean_word_length': 4.906532129580457, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Reconstruction of People in Loose Clothing

3D human body reconstruction from monocular images is an interesting and ill-posed problem in computer vision with wider applications in multiple domains. In this paper, we propose SHARP, a novel end-to-end trainable network that accurately recovers the detailed geometry and appearance of 3D people in loose clothing from a monocular image. We propose a sparse and efficient fusion of a parametric body prior with a non-parametric peeled depth map representation of clothed models. The parametric body prior constraints our model in two ways: first, the network retains geometrically consistent body parts that are not occluded by clothing, and second, it provides a body shape context that improves prediction of the peeled depth maps. This enables SHARP to recover fine-grained 3D geometrical details with just L1 losses on the 2D maps, given an input image. We evaluate SHARP on publicly available Cloth3D and THuman datasets and report superior performance to state-of-the-art approaches.


Introduction
Image-based modeling of loosely clothed 3D humans is an interesting and challenging open problem in computer vision. It has several applications in the domains of entertainment, AR/VR, sports, and healthcare. Traditional solutions based on stereo/multi-view reconstruction [13,9,11,15,39] require studio environments with multiple synchronized calibrated cameras, yet have limitations in recovering high-frequency geometrical details. Advancements in commercial depth sensing over the last decade [38,5,47,30] has partly helped in overcoming this limitation. Nevertheless, these sensors suffer from motion blur in dynamic scenes. Additionally, both RGB and depth camera-based methods are susceptible to severe self-occlusion artifacts caused by loose clothing and articulated body poses.
With the advent of deep learning models, significant interest has garnered around 3D reconstruction from a monocular image [20,42,16], which is an ill-posed problem.
Challenges like self-occlusions, viewpoint variations, and clothing obstacles make the scenario more difficult. One class of existing deep learning solutions attempts to fit a parametric body model like SMPL [25] to a monocular input image using global image features [20,14,32].Recently, several approaches [1,8,33,2] have proposed parametric clothing over the SMPL body. However, these methods can only accommodate relatively tight clothing styles as they primarily estimate local displacement/deformation of original SMPL vertices.
The other class of non-parametric body reconstruction techniques pose no such body prior constraints [36,37,29,42,7,44] and hence can potentially handle loose clothing scenarios. In particular, the recent implicit function learning models, PIFu [36] and PIFuHD [37] estimate voxel occupancy by utilizing pixel-aligned RGB image features computed by projecting 3D points onto the input image. However, the pixel-aligned features suffer from depth ambiguity as multiple 3D points are projected to the same pixel. As an alternate representation for 3D objects/scenes, some of the recent works model scenes as multiple (depth) plane images (MPIs) [41]. MPI is altogether a layered representation of depths, i.e. each layer is at a fixed depth in the camera frustum. 3D human body reconstruction has also been attempted in the same vein by predicting front and back depth maps in [12]. However, they fail to handle self-occlusions by body parts. Another interesting work [19] attempted to address the self-occlusion problem by predicting multiple peeled depth maps. Their sparse representation of the human body was obtained by encoding the 3D body surface using inverse ray-tracing formulation. Nevertheless, these approaches fail to take into account the fine-grain local geometry structure and do not seek to enforce global consistency on the body shape/pose to encourage physically plausible shapes and poses of human body parts.
The aforementioned problems can be addressed by introducing a body prior while reconstructing humans in loose clothing. The volume-to-volume translation network proposed in [48] attempts to fuse image features with the SMPL prior in a volumetric representation. However, volumetric convolutions are computationally expensive and is limited in resolution. Recently, ARCH [18] proposed to induce a human body prior by sampling points around a template SMPL mesh before evaluating occupancy labels for each point. However, sampling around the canonical body surface is not sufficient to reconstruct humans with complex articulated poses in loose clothing. Another interesting work [17] attempted to refine implicit function estimation by fusing volumetric features and pixel-aligned features to resolve local feature ambiguity. However, volumetric feature estimation is still computationally expensive and their method is not end-to-end trainable. Thus, existing methods either lack computational efficiency or compute only small deformations around the SMPL body prior. However, in loose clothing scenarios, garments can also undergo geometrical deformations that are independent of the underlying body shape and pose. Hence, there is an acute need to learn these deformations apart from body shape/pose deformation. This can be achieved by efficiently combining complementary strengths of parametric and non-parametric reconstruction paradigms.
In this paper, we propose an efficient, sparse, and robust 3D body reconstruction method that can successfully handle significantly loose clothing and skewed viewpoints. Our method combines the SMPL body prior with peeled depth predictions to achieve geometrically rich 3D body reconstruction from a monocular image. Starting with a good estimate of the SMPL body from a monocular image (e.g., using [40]), we encode this mesh into the sparse 2D peeled depth map representation proposed in [19] to obtain SMPL peeled depth maps. The SMPL prior along with the monocular RGB image is fed as input to our network which then predicts a dense additive residual deformation for each pixel in the SMPL peeled depth maps. This additive depth-based deformation accounts for clothing-specific geometrical deformations at each location near the body surface. However, such modeling do not focus on reconstructing regions where the clothing is far from the body surface. To reconstruct such regions, we propose to predict peeled depth maps in a separate prediction branch of our network. These predicted peeled maps are later fused with the residual deformation maps along with the initial SMPL peeled depth maps to obtain the final fused peeled depth maps. A separate decoder is used to predict peeled RGB maps. The peeled RGB maps are superimposed on the fused peeled depth maps to recover a colored point-cloud and subsequently a vertex colored mesh. The SMPL prior constrains our model in two ways: first, the network retains the geometrically consistent body parts that are not occluded by clothing, and second, it provides a body shape context that improves prediction of peeled depth maps. This enables our network to learn with only L 1 losses unlike other existing methods that use GAN loss and 3D Chamfer loss.
Our technical contributions are listed below: • We propose a novel end-to-end trainable encoderdecoder architecture our method, that uses only L1 losses on 2D maps for predicting a 3D body model with loose clothing from monocular image.
• Our method achieves sparse and efficient fusion of a parametric body prior with non-parametric peeled depth representation. • We evaluate our method on various publicly available datasets and report superior qualitative & quantitative results as compared to state-of-the-art methods.

Related Work
Parametric Body Fitting. Estimating the 3D human body pose [26,31] using deep neural networks has achieved great success with robust performance. The naked animatable 3D body is represented by SMPL [25], SMPL-X [34], SCAPE [3]. These models can be estimated from a single image by directly regressing the 3D joint locations. HMR [20] proposes to regress SMPL parameters when minimizing re-projection loss with the known 2D joints. Different priors have been used to refine the parametric estimates as in [43,32,23,21]. Despite these approaches being computationally efficient, they lack person-specific details. SMPL vertex offset estimates have been proposed to understand tight clothing details [8,45,24,46].

Non-parametric Body Reconstruction.
Recovering 3D human body from multi-camera setups employ voxel carving, triangulation, multi-view stereo, shape-from-X techniques [4,11,9,28]. With the advent of deep learning, initially voxel methods gained popularity as voxels are a natural extension to 2D pixels [44,42,48]. SiCloPe [29] estimates silhouettes in novel views to recover underlying 3D shape. Recent implicit function learning methods for human body reconstruction use locally-aligned pixel features space [36,37]. However, they suffer from sampling limitation and do not model explicit shape representation. Peeled maps [19] proposed a sparse representation by estimating only surface intersections by posing the problem as an extension to ray tracing. Similar idea is used for view synthesis by NERF [27] where it samples points along the camera ray to evaluate RGBσ on these samples.
Prior-based Non-Parametric Body Reconstruction. ARCH [18] combines the SMPL parametric body model with implicit functions to assign skinning weights for the reconstructed mesh. However, the method cannot reconstruct loose clothing in complex poses. Geo-PIFu [17] learns deep implicit functions by utilizing structure-aware 3D voxel features along with 2D pixel features. Dee-pHuman [48] leverages dense semantic representations from SMPL as an additional input. Nevertheless, both Geo-PIFu and DeepHuman are volumetric-regression based approaches that incur high computational costs.

Method
In this section, we first outline peeled representations for 3D shapes followed by the details of our proposed method.

Peeled Representation
Peeled representation is a sparse, non-parametric encoding of 3D shapes [19]. Each textured 3D body is represented as a set of 2D maps -four depth maps and four RGB maps. The depth and color values are recorded at each intersection of a camera ray with the 3D surface. This representation is more efficient than voxels and implicit functions as it only stores ray-surface intersection in a 2D multi-layered layout. Although this representation is able to handle severe selfocclusions, it lacks the ability to predict plausible human body shapes for complex poses as there is no inductive bias on the human body structure.

Method Overview
Given an image I of a person in arbitrary pose with loose clothing, we aim to reconstruct a textured 3D body surface with high-frequency geometrical details in clothing and plausible body part structure, as shown in Figure 2. There are three main components to our method i) An input peeled SMPL prior ii) Additive Residual Deformation (RD) prediction, and iii) Peeled map fusion. First, we get the SMPL body mesh fitted to the input image and convert it to a peeled depth representation to obtain a peeled shape prior D smpl as shown in Figure 2(b). This along with I is fed as input to the shared encoder in our network. The network then predicts three outputs through different decoder branches, namely, peeled RGB maps R, peeled depth maps D peel and peeled residual deformations D rd , as shown in Figure 2(c-e). The topmost decoder branch predicts only three peeled RGB maps as the input I naturally acts as the first peeled RGB map. The predicted peeled depth maps D peel and residual deformation maps D rd are subsequently combined using D smpl in a peeled map fusion module to get the final fused peeled depth maps D f used shown in Figure 2(f). Subsequently, a colored point-cloud representation is obtained by back-projecting D f used and R to 3D coordinates using known camera intrinsics, as shown in Figure 2(g). This point-cloud is further post-processed to obtain a mesh reconstruction using Poisson [22] method. A detailed discussion on these network components and training protocol including the loss functions is provided below.

Peeled Shape Prior
We estimate the SMPL model parameters (β, θ and global translation/rotation) for an input monocular image I using [40]. The predicted SMPL mesh is converted to sparse peeled depth map representation defined as: (1) The input image concatenated with the SMPL peeled depth maps D smpl forms the input to our network .

Residual Deformation (RD)
To estimate image specific deformations from the naked SMPL prior input, we propose to predict Residual Deformation maps D rd by computing pixel-wise offsets from the SMPL peeled depth maps D smpl , defined as: (2) More specifically, this models deformations of the SMPL prior in accordance with the clothing present in the input image. On pixels where there is no clothing (eg. face and hands), it predicts the offsets to be 0, thereby retaining the geometrical structure of body parts, as shown in Figure 3.

Peeled Map Fusion
The RD map modeling does not focus on regions where loose clothing is far from body surface. Hence, we propose to compute the final fused peeled depth maps D f used as the fusion of RD maps D rd and predicted peeled depth map D peel using the SMPL prior. The i th layer fused peeled map i.e., D i f used ∈ D f used is defined as: is element-wise multiplication and for each i th layer D i peel ∈ D peel , D i rd ∈ D rd and D i smpl ∈ D smpl .

Training
The learning objective is defined as: where λ rd , λ rgb and λ sm are regularization parameters. We provide the formulation for the individual loss terms below.
L peel captures the sum of L 1 norm between ground truth peeled depth map D i peel and predicted peeled depth map D i peel for each of the i th peeled map layer.
L rd constraints the RD predictions of SMPL to that of ground truth offsets. We also enforce per layer gradient smoothness of the predicted ( D i rd +D i smpl ) and ground truth (D i rd + D i smpl ).
Additionally, We also train our network with L 1 loss between predicted and ground truth RGB peeled maps (L rgb ).

Discussion on Architectural Choices
Existing prior-based non-parametric body reconstruction methods represent the body prior in 3D space (either voxels or meshes). However, this leads to computational inefficiencies as they need to densely probe the 3D space. Therefore, we adopted a sparse 2D peeled representation [19] of 3D shapes that can be efficiently processed by 2D convolution layers. Moreover, it enables to deform the body prior in order to approximate the loose clothing using residual deformation maps. Additionally, the peeled map representation also inherently addresses the self-occlusion issue.
In terms of network architecture, we use a shared encoder that encodes the peeled shape prior along with the input image using ResNet blocks into a shared latent representation. This enables a joint encoding of both geometry (coming from SMPL prior) and appearance and further regularizes the predicted peeled maps.
Regarding our peeled map fusion strategy, we model deformations of the SMPL prior surface as a pixel-wise residual offset in peeled layer representation. Unlike other existing methods ( [24,18,48]) that model 3D deformations directly (in either volumetric or surface representations), our method is more efficient as we treat it as a simple pixel-wise offset. Additionally, this enables a more effective and natural fusion of predicted geometry of loose clothing with that of the underlying body.

Implementation Details
Our multi-branch encoder-decoder network is trained end-to-end for 25 epochs. The shared encoder consists of 3 downsampling layers followed by 18 ResNet blocks. Sigmoid activation is used in last layer of the D peel and D rd decoder branches while a tanh activation is used for the R decoder branch. The D rd output values are scaled to a [−0.15, 0.15] range, which approximately maps to [−15cm, +15cm] metric scale range and holds empirical validity for cloth variations present in the datasets. Here, we use only 4 layers of peeled representation, although the method is generalized to more number of peeled layers.
We use the Adam optimizer with an exponentially decaying learning rate starting from 5 × 10 −4 . Our network takes 30 hrs to train on 4 Nvidia GTX 1080 Ti GPUs with a batch size of 8 and λ rd , λ rgb and λ sm are set to 1, 0.1 and 0.001, respectively. We use [10] for rendering the peeled maps.

Datasets
We perform both qualitative and quantitative evaluations on the following publicly datasets. Cloth3D [6] is a collection of 6500 sequences of draped SMPL meshes simulated with MoCap data. Each frame of a sequence contains garment and body-specific deformation information. Garments styles range from skirts to very loose robes. We augment this data by capturing SMPL texture maps with minimal clothing to simulate realistic body textures using [1]. For each sequence, five frames are randomly sampled and the naked body is subtracted from the garment as shown in Figure 4 to obtain clothed mesh sans occluded body as ground truth for training. We use the ground truth SMPL parameter to get initial peeled maps prior for our training. We also augment this data by rotating each mesh by 45 • , 60 • , −45 • along yaw axis to increase viewpoint variations.
THuman [48] consists of 6800 human meshes registered with SMPL body in varying poses and garments. The dataset was obtained using consumer RGBD sensors. Although the dataset has diverse poses and shapes, it has

Qualitative Results
Our method is able to predict high-fidelity human reconstructions with very loose clothing, as shown in Figure 5 and Figure 6. We demonstrate qualitative results for different clothing styles in Figure 6, where we were able to successfully reconstruct complex body poses(bottom row). Please refer to supplementary material for additional results.

Quantitative Evaluation
To quantitatively evaluate performance of our method, we compute Point-to-Surface (P2S) distance and Chamfer Distance (CD) using the point-cloud obtained from the predicted fused maps. P2S calculates the distance between this point-cloud and the ground truth mesh. CD calculates the distance between this point-cloud and the point-cloud sampled from the ground truth mesh.

Comparisons with the state-of-the-art
We evaluate the aforementioned metrics on Cloth3D dataset on PIFu [36], ARCH [18], GeoPIFu [17] and PeeledHuman [19]. Note that we retrained all these competing methods on Cloth3D data. We consider predicting  Figure 6: Qualitative results: Our method predicts consistent body parts along with realistic clothing deformations from a monocular input image. Note that these are per-frame reconstructions.
Unlike [19] that uses GAN with CD loss, we use a simple encoder-decoder architecture without CD loss on the predicted depth maps. To perform a fair comparison with ARCH, instead of sampling 3D points around the SMPL body in the canonical pose, we directly sample from the final pose for evaluation of labels. Similar to other methods, we transform all the predicted models from different methods to the canonical coordinates of the ground truth mesh. Table 1 summarizes the quantitative results where we outperform existing methods on Cloth3D data by a good margin for both CD and P2S metrics. Figure 7 shows that our method consistently outperforms other existing methods in terms of quality of reconstructed geometrical details around both body and cloth region, over varying body shape and clothing styles.
We also evaluate our method on THuman dataset with the evaluation metric code provided by [17] and reported results in Table 2  GeoPIFu as THuman dataset has shapes with primarily tight clothing scenarios in contrast to Cloth3D dataset, which has widely loose clothing styles, where we outperform GeoPIFu [17]. Finally, we show reconstruction results on DeepFashion [?] dataset in Figure 8.

Ablation Study
Impact of Peeled Map Fusion: In Figure 10, it can be observed that fusion of the RD maps with the predicted peeled maps helps to retain the body parts in complex and self-occluded poses. On the other hand, in the case of our baseline setup (without fusion module), we obtain distorted body parts.

Impact of Architecture Variations:
We also evaluate the performance of our method by varying number of ResNet blocks in encoder as shown in Table 3. We can observe that even with much smaller network size (6 ResNet blocks), our network is able to achieve better performance when compared to baseline setup and all other existing methods compared in Table 1.

Effect of Upsampling on the Predicted Point-cloud
The predicted point-cloud P, generated from the fused depth maps, can sometimes be sparse and contain holes. This issue arises because the perspective projection causes loss of information in the peeled maps, and this loss of information is reflected in the back-projected point-cloud.
To tackle this issue, we create a simplified point-cloud P sub by uniformly sampling ∼ 20, 000 points from P. Subsampling helps in avoiding the amplification of isolated noisy points, majorly from the parts that are not visible in the input image. Subsequently, we perform upsampling of P sub using [35] to obtain dense, upsampled point cloud P dense . The upsampling closes most of the potential holes but might lose the fine-grained geometrical details in P.
Hence, we superimpose P and P dense , which closes a significant number of holes and retains the geometrical details, as shown in Figure 9.

Conclusion
We introduced a novel shape-aware peeled representation for the reconstruction of human bodies with loose clothing. Our method achieves sparse and efficient fusion of parametric body prior with non-parametric peeled depth representation. We evaluated our method on various publicly available datasets and reported superior qualitative and quantitative results as compared to state-of-the-art methods. As part of future work it would be interesting to introduce temporal consistency in the proposed solution.