{'fraction_non_alphanumeric': 0.04101409305973737, 'fraction_numerical': 0.011284568479113428, 'mean_word_length': 4.8166298749080205, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
3D Magic Mirror: Clothing Reconstruction from a Single Image via a Causal Perspective

This research aims to study a self-supervised 3D clothing reconstruction method, which recovers the geometry shape, and texture of human clothing from a single 2D image. Compared with existing methods, we observe that three primary challenges remain: (1) the conventional template-based methods are limited to modeling non-rigid clothing objects, e.g., handbags and dresses, which are common in fashion images; (2) 3D ground-truth meshes of clothing are usually inaccessible due to annotation difficulties and time costs. (3) It remains challenging to simultaneously optimize four reconstruction factors, i.e., camera viewpoint, shape, texture, and illumination. The inherent ambiguity compromises the model training, such as the dilemma between a large shape with a remote camera or a small shape with a close camera. In an attempt to address the above limitations, we propose a causality-aware self-supervised learning method to adaptively reconstruct 3D non-rigid objects from 2D images without 3D annotations. In particular, to solve the inherent ambiguity among four implicit variables, i.e., camera position, shape, texture, and illumination, we study existing works and introduce an explainable structural causal map (SCM) to build our model. The proposed model structure follows the spirit of the causal map, which explicitly considers the prior template in the camera estimation and shape prediction. When optimization, the causality intervention tool, i.e., two expectation-maximization loops, is deeply embedded in our algorithm to (1) disentangle four encoders and (2) help the prior template update. Extensive experiments on two 2D fashion benchmarks, e.g., ATR, and Market-HQ, show that the proposed method could yield high-fidelity 3D reconstruction. Furthermore, we also verify the scalability of the proposed method on a fine-grained bird dataset, i.e., CUB.


INTRODUCTION
Nowadays, online shopping become popular [29]. People can purchase clothing via online shopping sites, e.g., Amazon and eBay. However, there remains a gap between the display images and the real product quality [6]. In an attempt to minimize such a visualization gap, we study the 3D clothing reconstruction from a single image in this paper. Given a 2D clothing image and the foreground mask, we intend to reconstruct a 3D mesh, which recovers the geometry shape, and texture of the target clothing. Besides, the clothing  Figure 1: Motivation. Here we compare the proposed approach with prevailing template-based methods, i.e., HMR [17] and ROMP [44] on a fashion dataset ATR [24]. We re-implement and visualize results with the color mapping according to the projected location. The first row is the front view, and the second row is the 3D mesh rotated with 45 • . The template-based model can capture the human poses but miss non-rigid objectives, such as hairs, handbags and dresses.
However, there remain three challenges. First, existing works [17,26,27] typically focus on human pose estimation and body reconstruction via parametric models, e.g., a morphable body template [31]. However, pre-defined body parameters usually are not scalable from the human body to the non-rigid clothing, e.g., dresses and handbags, largely losing fine-grained clothing details. As shown in Figure 1, we re-implement two prevailing methods, i.e., HMR [17] and ROMP [44], which both successfully capture the human pose but miss the cloth shape. In contrast, our methods leverage a deformable model to further facilitate learning non-rigid objects.
Second, 3D annotations are difficult to obtain due to the annotation difficulty and time costs. There are no public large-scale 3D clothing mesh datasets for supervised learning. In contrast, the availability of the large-scale 2D fashion datasets, such as ATR [24] and Market-1501 [56], makes training data-hungry deep-learned approaches become feasible. The success has been proved in the 2D computer vision tasks, including human parsing [16,25,30,41], person re-ID [14,50,51], attribute recognition [9,28,47] and pedestrian image generation [10,32,38,58]. With the recent development in self-supervised learning and deep-learned models, one straightforward idea is raised whether to leverage the 2D data for 3D reconstruction, even without the 3D annotations. This idea inspires us to explore the feasibility of self-supervised learning with causality design. The causality-aware structure helps the prior knowledge learning (e.g., learning the human prototype) during training. Here we also show the common example "Dating with Beautiful Girl" from The Book of Why [36] in (a), which is similar to our simplified dilemma on two variants, i.e., shape and camera pose in (b). Since we study an inverse problem, we also draw (c) and (d). We note that there is a compensation effect when we estimate the posterior probability. As shown in (d), given the observed foreground shape, the model needs to estimate the posterior 3D shape and posterior camera pose. There are two possible alternatives for the network to learn, i.e., a big shape from a long distance or a small shape from a short distance. Therefore, it makes the model difficult to converge an answer.
Third, one scientific question still remains in the single image 3D reconstruction. The primary implicit variables for reconstruction are camera viewpoint, shape, texture, and illumination. Ideally, these four factors are independent of each other. However, it remains challenging to disentangle the implicit variables in practice. During implementation, one typical dilemma is the ambiguity between the camera and shape [23]. For example, given one 2D image, it is hard to decide the size of the target. There are two possible answers. One large object is far from the camera or one small object is close to the camera. Despite different original sizes, the two objects have the same projection size in the photos. In this paper, we introduce the structure causal map [35], which is widely used in causal inference. It helps us to identify the problem as a "collider" connection and we show a simplified sample of two variants in Figure 2. Following the spirit of the causal map ( Figure 3), we discuss the failure reason and leverage the "intervention" tool, i.e., two expectation-maximization loops, to avoid this "collider" dilemma. (1) Encoder Loop. A key underpinning the factor disentanglement is using independent encoders for different implicit variables. The weights of encoders are independent of each other. Despite the disentangle design in the network forwarding, we observe that, if the reconstruction fails, all four encoders are usually penalized equally. It largely impacts the single encoder training. The phenomenon that the loss of the "result" impacts the "cause" encoder is the compensation effect in "collider". For instance, even if the texture encoder is satisfied, the reconstruction failure in the 3D shape also leads to a large punishment on the texture encoder. Therefore, in this work, we propose an expectation-maximization loop to "free" one factor while "intervening" other factors. We first obtain the three out of four factor expectation, and then maximize the likelihood of the rest one factor. In this way, we find that the loss propagation can effectively penalize the worst encoder. It also helps the encoder disentanglement from the loss back-propagation aspect. (2) Prototype Loop. We introduce a fixed-size prototype shape as previous works [13,21,23] and update the prototype shape during training. The general assumption is that the shapes of different instances are close to the prototype shape. Therefore, the shape changes are limited around the prototype. By the prototype design, we physically free the shape encoder from estimating the vertex movement due to object size changes. Therefore, the camera encoder is fully responsible to estimate the scale changes by adjusting the camera distance prediction. We can regard the prototype template as a latent variable. During training, the prototype is generated from the average output shape in the whole dataset, which can be viewed as an expectation step (E-step). Then updating both the shape encoder and camera encoder is a maximization step (M-step), which maximizes the parameter likelihood based on the prototype expectation. This loop further disentangles the shape encoder and camera encoder and makes the shape encoder focus on learning intra-class variants, e.g., instance deformation, rather than the global camera distance changes.
In an attempt to overcome the above-mentioned challenges, this paper proposes Causality-aware Self-supervised Learning (CASL) to adaptively reconstruct the input 2D clothing image without the 3D mesh annotation. We study the existing works (see Figure 3) and introduce an explainable structural causal map (SCM) to build our model and guide the optimization strategy. (1) Following the spirit of the causal map, we deploy four independent 3D attribute encoders and a differentiable render for reconstruction. The encoders are to extract 3D attributes from 2D images and foreground masks, including camera viewpoint, geometric shape, texture, and illumination. Then the attributes are fed to the differentiable render to reconstruct the 3D mesh. Different from existing works [13,23], we explicitly introduce the prior template to help the camera estimation and shape offsets estimation, which is aligned with human observation. If we foreknow the human or bird prototype, it helps us to predict the camera position as well as the intra-class variant (such as leg movement). (2) We leverage the causality intervention tool, i.e., two Expectation-Maximization Loops, to help the prototype learn and disentangle encoders from the confusing loss punishment. Extensive experiments verify the effectiveness of the proposed method on clothing reconstruction. Besides, we observe that the proposed method also has the potential to improve the reconstruction performance of general objects, like birds in the fine-grained CUB dataset [46].
To summarize, our contributions are three-folds: • We identify the three challenging problems in the 3D clothing reconstruction: 1) Non-rigid objects; 2) No 3D annotations; 3) Reconstruction ambiguity. • In an attempt to solve these challenges, we propose a selfsupervised learning method with causality design to reconstruct the 3D clothing mesh from large-scale 2D image datasets. Following the spirit of the causal map, we re-design the encoder structure and leverage the "intervention" tool, i.e., two expectation-maximization loops, to facilitate the 3D attribute encoder learning. • Extensive experiments on two fashion datasets verify the effectiveness of the proposed method both quantitatively and qualitatively. Furthermore, the experiments on the finegrained bird dataset also show that the proposed method has good scalability to other reconstruction tasks.  . We compare the proposed method with three typical 3D reconstruction works, including CSM [21], UMR [23] and SMR [13]. Here we show the image reconstruction loop, i.e., "2D → 3D → 2D". (a) Given one 2D image and the foreground mask, CSM only applies two encoders for Camera and Texture, since it assumes that the basic shape is shared, ignoring intra-class changes. (b,c) UMR and SMR further introduce the shape encoder, which is to predict the shape offset. The final shape is obtained by adding the shape offset to the shape template. (d) In this work, we argue that the predicted shape offsets are also conditioned on the prior template. Besides, the prior template also impacts the camera prediction. Therefore, we explicitly introduce the dependency with the prior prototype. Besides, it is worth noting that our prototype is initialized from a sphere and iteratively updated during training, which does not require any hand-crafted initialization. (Qualitative comparison with more other methods is in Table 1 ✓ ✓ ✓ Texformer [52] ✓ ✓ ✓ SMR [13] ✓ Ours ✓ Table 1: Comparison with existing methods in terms of supervisions. The proposed method harnesses relatively weak supervision for 3D reconstruction from a single image. * : the method deploys the manual template for initialization. † : the viewpoint annotation is optional. It is also worth noting that some works take 2D input images with blank white background as inputs, and we also view this line of works deploys the foreground mask.

3D Reconstruction from Single Image
Human can estimate the 3D structure from one single image. Many works deploy a parameter-based template [17,27,44], which is robust but also limits the representative ability to non-rigid objects. To enable more degrees of freedom, Deephuman [59] adopts a U-Net voxel prediction method to reconstruct the human body and clothing, yielding excellent performance, but relatively dense depth and ground-truth 3D annotations are needed. To reduce the dependency on the 3D annotations, PrGANs [8] train a generative adversarial network (GAN) to generate the 3D voxel model (from scratch) from lots of 2D images with different viewpoints. To further avoid the dense prediction in the voxel format, Pixel2Mesh [48] trains a graph neural work with a convolutional neural network encoder to reconstruct 3D mesh by deforming a sphere. We note that Pixel2Mesh is a fully supervised learning work. The ground-truth mesh from ShapeNet [4] is used. Lots of regularization losses, e.g., laplacian regularization and flatten regularization, have been proposed, which are widely used in following works. With the spirit in Pixel2Mesh, lots of researchers have explored weaker supervision, i.e., self-supervised learning. VPL [19] leverages the viewpoint annotation to ensure the reconstruction consistency between different views. However, the annotated viewpoint-image pairs are hard to acquire or be simulated for living objects, such as birds or humans. To invade the viewpoint annotation, one of the early works is Canonical Surface Mapping (CSM) [21], which treats 3D reconstruction as a dense key-point estimation problem and applies the cycle consistency to regularize the model learning. However, CSM deploys a fixed pre-defined shape template, which largely limits intra-class shape changes for different instances. To address the shape limitation, CMR [18] first proposes to use a learnable shape template and Li et al. [23] further propose UMR adopt a two-stage training strategy for template updating. The model is first trained to obtain one category-aware template and then the second stage fine-tunes the learned mesh for every instance. The segmentation parsing [15] is also used in UMR for better alignment.
The contemporary work, IMR [45], first introduces the mapping function instead of the vertex location regression, saving computation costs due to a large number of vertices. Taking a step further, SMR [13] is proposed to better align the 3D mesh by mix-up and conduct the auxiliary vertex classification task. The proposed method is mainly different from existing work in three aspects: (1) Weaker supervision. As shown in Table 1, the proposed method demands limited supervision and mainly leverages the large-scale multi-view images to learn the prior knowledge. (2) Model design. As shown in Figure 4, the network design follows the causal map. In particular, the model contains four independent encoders for four implicit factors, i.e., shape, camera pose, illumination, and texture. Besides, we explicitly take more dependencies, e.g., 3D prior template, into consideration. For instance, both shape encoder and camera encoder harness the integration module (IM) to introduce the 3D prior knowledge. (3) Optimization strategy. To deal with the compensation effect in the loss punishment, we deploy the "intervention" tool, i.e., two expectation-maximization loops, to facilitate the 3D attribute encoder learning.

Causal Learning
Causal learning is to identify causalities from a set of empirical factors, which can be either pure observations or outcomes, e.g., changes and interventions [37]. To represent the causalities, a causal model is usually defined via structural equations and graphs. For instance, the structural causal model proposed by Pearl et al. [35] is a combination of directed acyclic graphs (DAGs). According to the structural causal model, manipulations can be conducted to optimize the estimated relations between variables, e.g., "Do" operation is to cut certain directed edges and control the target variable [35]. Different from data-driven deep learning approaches, the causality methods intend to discover and leverage the causal relation between entities. For example, given data on the chocolate preferences of Nobel Prize winners, deeply-learned models are prone to conclude that the more chocolate a person eats, the more likely he or she is to win a Nobel Prize. Data-driven methods only see the correlation between phenomenons, while causal learning identifies the shared cause, i.e., richer people both eat more chocolates and win more Nobel Prizes. Several existing works also explore implicit causal learning to discover the causal factors during training. CausalGAN [20] trains a generator, which is consistent with an implicit causal graph, and is able to sample from either conditional labels or interventional distributions. Similarly, CausalVAE [54] is a VAE-based causal framework to learn disentangled representations. CausalVAE introduces a Structural Causal Model layer, which discovers latent causal factors in data with graph constraints. Both methods implicitly harness causal mapping by learning latent code or adding one graph constraint loss. However, the causality learned from data is not always accurate and explainable, limiting the causality effect.
Different from CausalGAN and CausalVAE, our work explicitly builds the model according to the structural causal map. In particular, our model follows the spirit of causality between semantic entities to (1) explicitly consider the causality relation between entities, e.g., leveraging the prior template to help both camera encoder and shape encoder learning; and (2) explicitly apply the "intervention" tools to solve the ambiguity of learning multiple variables, e.g., camera, texture, shape, and illumination.

Expectation Maximization
Expectation Maximization (EM) is an iterative method to find the parameters with maximum likelihood in statistical models [33]. The EM algorithm iteratively conducts two kinds of steps: an expectation step (E-step) to obtain the expectation of latent variables and a maximization step (M-step) which computes parameters to maximize the expected likelihood based on the latent variables. Since M-step updates parameters, it affects the E-step in the next round. In this way, the EM algorithm can keep updating until the convergence. The EM algorithm is usually applied to scenarios that miss the observation of implicit variables, such as Gaussian mixture model [53].
For 3D reconstruction, we also meet a similar problem, when we need to estimate four reconstruction factors simultaneously, i.e., camera position, shape, texture, and illumination from a single 2D  Figure 4: Overview. Here we show a "2D→3D→2D" loop. We follow the causal map in Figure 3 (d) to design the pipeline. Given one pair of clothing image and the mask , we deploy four independent encoders , , , and for camera position, shape, texture and light estimation. We introduce the integration module (IM) to fuse the local feature from 3D prior template¯. Then we apply the colorize function to obtain the mesh, and utilize the render to re-project the mesh to 2D space via the project function . Finally, we obtain the reconstructed andˆ. During inference, we could manipulate intermediate camera attributes to generate novel-view images of the target person.
image. It is worth noting that if we have three out of four factors, we can easily optimize the rest by efficient gradient backward. This is an M step. Inspired by the expectation-maximization algorithm, we can fix the encoders (E-step) to conduct factor predictions, e.g., camera position, texture, and illumination, and then maximize the likelihood of the rest variable, e.g., shape (M-step). If readers are familiar with the causality, this process actually is a "Do" operation in the causal map [36]. In the E step, we remove the arrows that target camera pose, illumination, and texture, but leave one arrow for 3D Shape Offsets. In the M-step, we back-propagate various losses from the reconstructed image, and only the parameters of the shape encoder are updated (maximize the likelihood). In practice, we iteratively fix three out of four encoders and train the rest one encoder.

Overview
Given a clothing image and the foreground mask , we aim to infer the corresponding 3D mesh with texture (see Figure 4).
∈ [1, ] and is the number of the samples in the dataset. We do not require any extra 3D annotations. In this work, different from CausalGAN [20] and CausalVAE [54], we explicitly follow the structural causal map in Figure 3(d) to build the whole pipeline. Generally, the spirit of causality helps us to (1) disentangle the 3D representation from inherent correlations, such as the ambiguity between the shape and camera encoder, (2) re-consider the causal relation between entities, such as the 3D prior template (prototype) and camera position estimation. Specifically, we deploys four independent encoders, i.e., shape encoder , camera pose encoder , illumination encoder and texture encoder . The design follows the structural causal map. The decoder is based on the differentiable render [7], which does not contain any learnable parameters. Therefore, we can also regard the render as a fixed decoder. Following existing works [21,23], we also introduce a 3D prototypē ∈ R |¯|×3 , which explicitly involves the prior body structure to the network learning. The 3D prior template can be initialized with an arbitrary mesh. Without loss of generality, we apply the sphere shape (contains 642 vertices and 1280 faces) to initialize the 3D prior template. Here we set 642 vertices as the default setting to illustrate the proposed approach if not specified. During training, we keep updating the prior template¯. When inference, the model can deploy the latest 3D prior template (prototype) for testing.

Model Structure
Shape Encoder. We follow the causal map to explicitly introduce the 3D prior into the encoder learning. Given the input image-mask pair , and the 3D prior template¯, the shape encoder estimates the offsets Δ ∈ R 642×3 for every vertex: The final 3D shape is the sum of the 3D prior template and the 3D per-vertex offsets, and ∈ R 642×3 . Different from most existing works [13,23], which independently estimates the Δ from the input image and the mask , our shape encoder explicitly takes the prior template into the deformation prediction as ( , ,¯).
The main idea is straightforward, since predicting shape offsets depends on foreknowing the shape prior. We explicitly provide the shape template to help the training. In particular, the shape encoder contains one convolutional neural network (CNN) as backbone, one integration module (IM) and a fully connected layer (fc). Integration Module. As shown in Figure 5, we fuse the visual feature from both the input image/mask and the 3D prior template. We harness the integration module (IM) to extract the local visual feature according to the 2D location by projecting the 3D template to the X-Y plane. On the other hand, the global feature is generated by averaging the input visual feature (by global average pooling) and then we repeat the aggregated feature as the original size. The final Δ is predicted by a fully-connected layer on the concatenated feature of both global features and local features. Camera Pose Encoder. Similarly, the camera pose estimation also depends on the shape prior and the image. However, the previous works usually ignore the causal dependency on the shape prior¯. When people infer the object position, e.g., distance, azimuth, and elevation, it is necessary to foreknow the general object shape (general size, general shape, symmetry to which axis). Therefore, we also deploy one basic convolutional neural network (CNN) followed by an integration module (IM) and fully-connected layers as the camera position encoder. The camera pose encoder can be formulated as: where contains four factors, i.e., distance (1-dim), azimuth (1dim), elevation (1-dim), and X-Y position offset (2-dim). "dim" is dimension. The distance is also formulated as object scale in other works [23], while the azimuth is called as the rotation degree. We  Figure 5: Integration Module (IM). We harness IM to explicitly fuse the prior spatial information from template¯and the visual feature from ( , ). The visual feature is the output of the backbone, e.g., HR-Net [55]. As the arrow direction, we leverage the grid sampler to obtain the local feature from the corresponding X-Y of the visual feature. We concatenate global features and local features as final outputs.
notice that several existing works [13] do not include X-Y position, since they assume that the object is in the middle of 2D images. However, we find that including X-Y position offset also improves the robustness of the network during both training and testing time.
In the experiment, we further visualize the distribution of the learned four camera factors in Figure 9. Illumination Encoder. The illumination encoder is to regress the illumination direction and strength, which can be simply formulated as a 9-channel Spherical Harmonics coefficient [5]. Therefore, we adopt a basic convolutional neural network followed by a 9-channel fully-connected layer to predict the illumination vector from the input image-mask pair: Texture Encoder. In this work, we do not predict the color for every vertex. We follow existing works [18,21] to learn a texture flow as the UV map by a U-Net structure [40]. Given the input image and the corresponding foreground mask , we first predict the texture flow and then map the color according to the spatial location.
Decoder (Render). Finally, the decoder, i.e., render, can reconstruct the 3D mesh with color by simply combining the shape and . If we want to re-project the mesh to the 2D space, we further need the camera pose and the illumination direction . Therefore, the reconstructed imageˆcan be written as: where is the function to colorize the 3D mesh with the UV map . denotes the projection function mapping the mesh through camera parameters with the illumination . As a side product, we can also obtain the reconstructed foreground maskˆduring projection. We note that both and are based on the physical mapping, so there do not contain any learnable parameters.

Optimization Objectives
Image Reconstruction Loss. As shown in the right part of Figure 6, we calculate the pixel level 1 loss of the foreground area between the reconstructed image and the input 2D image: where ⊙ denotes element-wise multiplication, and E denotes the expectation.ˆandˆare the reconstructed image and mask projected from the 3D mesh. We note that the L focuses on the low-level input. Sometimes the generation quality is good but with small position shifts. To further ensure the generation quality from high-level activations, we also introduce the adversarial loss: where ⊕ means concatenation. For instance, ⊕ is a 4 channel input. denotes a multi-layer discriminator to classify whether the input is real or generated from our render. In practice, we adopt a basic WGAN structure [1] as the discriminator.
Otherwise, we introduce the IoU loss to compare the overlapping area between the generated mask with the ground-truth input mask.
Attribute Reconstruction Loss. As shown in the left part of Figure 6, we also conduct the 3D attribute reconstruction to ensure that the encoder and the decoder is self-consistent: The 3D attributes predicted from the reconstructed imageˆshould be the same as the predicted attribute from . Mesh Regularization.
(1) Laplacian loss [48] is a regularization to prevent self-intersection of mesh faces. It encourages adjacent vertices to move in the same direction, consequently, avoiding the local part of the mesh producing outrageous deformation. For each vertex position in the mesh shape , the laplacian coordinate is = − ∈ ( ) | | ( ) | | , where ( ) are the neighbor vertices of with connected edges. Specifically, the laplacian loss can be defined as L = E[||ˆ− || 2 2 ], whereˆand are laplacian coordinates of a vertex before and after the updation respectively; (2) Flatten loss is another regularization for keeping faces from intersecting [48]. The cosine of the angle between two adjacent faces is calculated. The flatten loss is defined as L = E[( (Δ ) + 1) 2 ], where Δ is the angle between two adjacent faces. The angle around 180 • implies a smooth mesh surface; (3) Symmetry loss constrains mesh deformations to be reflectional symmetric in the depth [45]. It can be expressed as L = E[|| ( ) + (˜)|| 1 ], where denotes the depth of the vertex and˜is the reflected vertex of ; (4) Deformation loss [18,23] is a regularization to discourage the mesh to deform excessively and to facilitate learning a meaningful average shape. It is defined as L = E[||Δ || 2 ]. Total Loss. We train four encoders and discriminator to optimize the total objective, which is a weighted sum of above-mentioned losses:  Figure 6: Losses. Here we show three kinds of losses on the causal map, which are the image reconstruction loss, the attribute reconstruction loss and the mesh regularization.

Optimization Strategy
Encoder Loop. During the implementation, we notice one main challenge is simultaneously optimizing the four encoders. The problem mainly lies in the image reconstruction loss. For instance, even if three out of four encoders provide correct prediction, the rest provides the wrong attribute, such as incorrect shape offsets. All four encoders are penalized equally. This is one typical "collider" case in causality. Therefore, one straightforward idea is to train one encoder (predict one cause) while fixing the other three encoders (control other causes). In this way, we can effectively penalize the target encoder. In particular, we adopt one expectation-maximization loop, which also is an "invention" tool in causal learning. For example, we fix the three encoders, e.g., , , , which cuts the arrows from inputs to three attributes, i.e., , and . Only the shape attribute still keeps the dependency from the input image-mask pair. Hence, when the loss is back-propagating, only the shape encoder is penalized. In this way, we disentangle four encoders not only in the forward passing design (independent encoder weights) but also in the loss of back-propagation. Prototype Loop. We also observe a common ambiguity between prototype updating and shape estimation. The problem is mostly due to Eq. 2. Since it is an addition equation, during gradient backpropagation,¯and Δ receive the punishment equally. It is hard to distinguish¯from Δ . Therefore, we adopt the causal invention tool, i.e., Expectation-Maximization. During training, we fix the prototype (control one cause) and maximize the shape offsets likelihood (predict another cause). After every training epoch, we leverage the mean shape offsets to update the¯=¯+ E[Δ ]. In practice, different from existing works (e.g., two-stage training [23]), we adopt a linear warming-up strategy [11] to update prototype slowly in the early epochs and harness the exception handling by clipping extreme deformations. In this way, we disentangle the prototype updating from the shape offsets estimation and learn the model in one go.

EXPERIMENT
We evaluate the proposed approach on two fashion datasets, ATR [24] and Market-HQ [56], and one widely-used bird dataset CUB [46]. Evaluation Metric. Since there are no ground-truth 3D meshes, we follow existing works [13] and adopt 2D metric, e.g., FID [12], SSIM [49] and MaskIoU, to evaluate projected images. FID compares the distribution of two sets of images. We denote the 3D reconstruction results as , the generated image with different viewpoints as and the side-view image as 90 . We follow [13] to rotate the camera viewpoint to save images for . SSIM provides a low-level similarity comparison between reconstructed images and input images. It reflects the effectiveness of the process "2D → 3D → 2D". Besides, we also apply the Mask-IoU to compare overlapping regions between the ground-truth mask and the foreground area projected from reconstructed 3D meshes. Reproducibility. Our code is based on Pytorch [34]. More training details and model structures are provided in Supplementary Material.

Quantitative Experiments
Comparison with Template-based Methods. We first compare with the off-the-shelf template-based methods [17,44] in Table 2. This line of methods is based on the body template with great robustness, but is not well scalable to the non-rigid clothing. Since no texture mapping function is built in template-based methods, we focus on comparing MaskIoU (%), which reflects the "2D → 3D → 2D" shape reconstruction quality. We observe that the proposed method achieves a higher MaskIoU score of 82.0% on the test set. The result is also consistent with the visualization in Figure 1. For clothing reconstruction, the proposed method is more scalable than template-based methods, covering more regions of interest. Comparison with Single-image Reconstruction Methods. As shown in Table 3, we compare the proposed method with other state-ofthe-art approaches [5,13,18,22,23] on the CUB dataset. Among existing works, SMR [13] has achieved the high-fidelity reconstruction and novel-view generation performance. In contrast, the proposed method yields a competitive reconstruction performance (84.0% MaskIoU and 82.5% SSIM) with SMR. At meantime, for novel-view generation, ours surpasses SMR with 76.2 FID .

Qualitative Experiments
Reconstruction and Novel-view Results . As shown in Figure 7, we reconstruct the person with non-rigid clothing. We could observe that the model not only successfully learns the legs and arms, but also captures non-rigid objects, including hair, dress and handbag. Since we disentangle four encoders, the proposed method could easily manipulate 3D attributes, including distance, elevation, shape and texture, for customization.
Exchanging Clothing. Inspired by 2D GAN-based work [58], we also show the result of changing the texture of any two persons but with a 3D mesh manner (see Figure 8). In particular, we apply the shape encoder and the texture encoder to extract the shape and the UV texture map . Then we deploy the render to generate the new mesh based on and . The first row and the first column are the input RGB images. The rest is the projected results of the new 3D meshes. We rotate the mesh for better 3D visualization. It verifies the robustness of the method. The learned UV map could successfully be aligned on different human meshes, even though we have not introduced any part annotations during the training process.

Ablation Study and Further Discussion
Camera Attribute Distribution. We observe that the camera encoder successfully captures the camera distribution in the Market-HQ test set. As shown in Figure 9, most samples are 2 ∼ 4 unit distances from the camera, and most persons are in the center of the figure with 0 X-Offsets and 0 Y-Offsets. Most samples appear in 0 • , 180 • or -180 • , which means that most people are facing towards or backward to the camera. It is aligned with the dataset setup since cameras are set up in front of the supermarket entrance or exit. The elevation of most samples is from -10 to 10, which is also aligned with the data collection setup, i.e., six horizontal-view cameras. Besides, we also show the distribution of mean shape offset Δ . We observe that most deformations are relatively small since we have introduced the 3D prior template. In a summary, the learned attribute statistics also verify that we disentangle the camera hyper-parameter, e.g., scale changes and position offsets, from the shape encoder. Does the two expectation-maximization loops help the encoder learning? Yes. As shown in Table 4, we conduct two ablation studies on Market-HQ. (1) One is to stop the prototype updating, i.e., No Prototype Loop, and we deploy the fixed sphere as the basic shape. It directly limits the shape deformation, compromising the reconstruction performance. (2) Besides, we also explore training all encoders simultaneously without encoder loop as "No Encoder Loop". We observe that the model can easily over-fit the front-view reconstruction quality but it does not perform well in the novel view, especially we look at the 3D mesh from the side view, i.e., 90 • . Does the integration module work? Yes. As one minor contribution, we design the integration module to explicitly fuse the prior prototype as local features for learning shape. As shown in Table 4,  removing integration module, i.e., No IM, leads to a performance drop in both reconstruction and novel-view generation. Limitation. There are two faces or two backs of heads on one reconstructed mesh. It is because our work is still based on a single image, and the learned model only "sees" one single view of the human. Especially on the ATR dataset (most photos are frontal faces), it can not learn the 3D prior, i.e., one person only has one face and one back of the head. Therefore, even if we introduce the discriminator from WGAN [1] on the 2D shape and texture, it can not provide 3D-aware adversarial loss. The model still largely relies on the symmetric structure to generate the back view. Hence, we Here we show the distribution of azimuths, distances, elevations, Offsets-X and Offsets-Y. Besides, we also provide the distribution of the mean shape offset Δ over the test set.  Table 4: Ablation Study on Market-HQ. "No IM" denotes that we remove the integration module. We observe that although "No Encoder Loop" leads the model to over-fitting the frontview reconstruction, the side-view performance FID 90 is extremely poor. In contrast, the full model takes a balance point between reconstruction and novel-view generation.
think that, in the future, the large-scale multi-view image datasets may help to further solve this limitation upon our work.

CONCLUSION
In this work, we study the 3D clothing reconstruction problem to build a "3D Magic Mirror". We notice that there are three primary challenges, i.e., non-rigid clothing, no 3D ground-truth annotations, and the inherent reconstruction ambiguity. In an attempt to solve the three problems, we study the self-supervised learning method with causality design to reconstruct 3D clothing mesh from large-scale 2D image datasets. Specifically, we follow the spirit of the structural causal map to re-design the output dependency, and leverage two expectation-maximization loops to facilitate the training process. In the experiment, we observe that, despite using relatively weak supervision, the proposed method is still competitive with other existing works, and shows great scalability to different datasets. In the future, we will continue to explore the application to 3D object re-id [60] and building reconstruction [57].