{'fraction_non_alphanumeric': 0.033805525760994194, 'fraction_numerical': 0.013906868365573104, 'mean_word_length': 4.539438639623028, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Escaping Plato's Cave: 3D Shape From Adversarial Rendering

We introduce PlatonicGAN to discover the 3D structure of an object class from an unstructured collection of 2D images, i.e., where no relation between photos is known, except that they are showing instances of the same category. The key idea is to train a deep neural network to generate 3D shapes which, when rendered to images, are indistinguishable from ground truth images (for a discriminator) under various camera poses. Discriminating 2D images instead of 3D shapes allows tapping into unstructured 2D photo collections instead of relying on curated (e.g., aligned, annotated, etc.) 3D data sets. To establish constraints between 2D image observation and their 3D interpretation, we suggest a family of rendering layers that are effectively differentiable. This family includes visual hull, absorption-only (akin to x-ray), and emission-absorption. We can successfully reconstruct 3D shapes from unstructured 2D images and extensively evaluate PlatonicGAN on a range of synthetic and real data sets achieving consistent improvements over baseline methods. We further show that PlatonicGAN can be combined with 3D supervision to improve on and in some cases even surpass the quality of 3D-supervised methods.

While methods exist to learn the 3D structure of classes of objects, they typically require 3D data as input. Regrettably, such 3D data is difficult to acquire, in particular for the "long tail" of exotic classes: ShapeNet might have chair, but it does not have chanterelle.
Addressing this problem, we suggest a method to learn 3D structure from 2D images only (Fig. 1). Reasoning about the 3D structure from 2D observations without assuming anything about their relation is challenging as illustrated by Plato's Allegory of the Cave [34]: How can we hope to understand higher dimensions from only ever seeing projections? If multiple views (maybe only two [40,13]) of the same object are available, multi-view analysis without 3D supervision has been successful. Regrettably, most photo collections do not come in this form but are now and will remain unstructured: they show random instances under random pose, uncalibrated lighting in unknown relations, and multiple views of the same objects are not available.
Our first main contribution (Sec. 3) is to use adversarial training of a 3D generator with a discriminator that operates exclusively on widely available unstructured collections of 2D images, which we call platonic discriminator. Here, during training, the generator produces a 3D shape that is projected (rendered) to 2D and presented to the 2D Platonic discriminator. Making a connection between the 3D generator and the 2D discriminator, our second key contribution, is enabled by a family of rendering layers that can account for occlusion and color (Sec. 4). These layers do not need any   Figure 2. Overview: We encode a 2D input image using an encoder E into a latent code z and feed it to a generator G to produce a 3D volume. This 3D volume is inserted into a rendering layer R to produce a 2D rendered image which is presented to a discriminator D. The rendering layer is controlled by an image formation model: visual hull (VH), absorption-only (AO) or emission-absorption (EA) and view sampling. The discriminator D is trained to distinguish such rendered imagery from an unstructured 2D photo collection, i. e., images of the same class of objects, but not necessarily having repeated instances, view or lighting and with no assumptions about their relation (e.g., annotated feature points, view specifications).
learnable parameters and allow for backpropagation [26]. From these two key blocks we construct a system that learns the 3D shapes of common classes such as chairs and cars, but also exotic classes from unstructured 2D photo collections. We demonstrate 3D reconstruction from a single 2D image as a key application (Sec. 5). While recent works focus on using as little explicit supervision [17,19,8,29,28,11] as possible, they all rely on either annotations, 3D templates, known camera poses, specific views or multi-view images during training. Our approach takes it a step further by receiving no such supervision, see Tbl. 1.

Related Work
Several papers suggest (adversarial) learning using 3D voxel representations [37,36,12,24,11,32,31,35,39,30,20] or point cloud input [1,10]. The general design of such networks is based on an encoder that generates a latent code which is then fed into a generator to produce a 3D representation (i. e., a voxel grid). A 3D discriminator now analyzes samples both from the generator and from the ground truth distribution. Note that this procedure requires 3D supervision, i. e., is limited by the type and size of the 3D data set such as ShapeNet [5].
Girdhar et al. [12] work on a joint embedding of 3D voxels and 2D images, but still require 3D voxelizations as input. Fan et al. [9] produce points from 2D images, but similarly with 3D data as training input. Gadelhan et al. [11] use 2D visual hull images to train a generative 3D model. Cho et al.'s recursive design takes multiple images as input [6] while also being trained on 3D data. Kar et al. [18] propose a simple "unprojection" network component to establish a relation between 2D pixels and 3D voxels but without resolving occlusion and again with 3D supervision.
Cashman and Fitzgibbon [4] and later Carreira et al. [3] or Kanazawa et al. [17] use correspondence to 3D templates across segmentation-or correspondence-labeled 2D image data sets to reconstruct 3D shapes. These present stunning results, for example on animals, but at the opposite end of a spectrum of manual human supervision, in which our approach receives no such supervision.
Closer to our approach is Rezende et al. [25] that also learn 3D representations from single images. However, they make use of a partially differentiable renderer [22] that is limited to surface orientation and shading, while our formulation can resolve both occlusion from the camera and appearance. Also, their representation of the 3D volume is a latent one, that is, it has no immediate physical interpretation that is required in practice, e. g., for measurements, to run simulations such as renderings or 3D printing. This choice of having a deep representation of the 3D world is shared by Eslami et al. [8]. Tulsiani et al. [29] reconstruct 3D shape supervised by multiple 2D images of the same object with known view transformations at learning time. Tulsiani et al. [28] take it a step further and require no knowledge about the camera pose, but still require multiple images of the same object at training time. They have investigated modelling image formation as sums of voxel occupancies to predict termination depth. We use a GAN to train on photo collections which typically only show one view of each instance. Closest to our work is Gadelha et al. [11] which operates on an unstructured set of visual hull images but receives three sources of supervision: view information gets explicitly encoded as a dimension in the latent vector; views come from a manually-chosen 1D subspace (circle); and there are only 8 discrete views. We take the image formation a step further to support absorption-only and emission-absorption image formation, allowing to learn from real photos and do so on unstructured collections from-the-wild where no view supervision is available.
While early suggestions how to extend differentiable renderers to polygonal meshes exist, they are limited to deformation of a pre-defined template [19]. We work with voxels, which can express arbitrary topology, e. g., we can generate chairs with drastically different layout, which are not a mere deformation of a base shape.
Similarly, inter-view constraints can be used to learn depth maps [40,13] using reprojection constraints: If the depth label is correct, reprojecting one image into the other view has to produce the other image. Our method does not learn a single depth map but a full voxel grid and allows principled handling of occlusions.
A generalization from visual hull maps to full 3D scenes is discussed by Yan et al. [38]. Instead of a 3D loss, they employ a simple projection along major axis allowing to use a 2D loss. However, multiple 2D images of the same object are required. In practice this is achieved by rendering the 3D shape into 2D images from multiple views. This makes two assumptions: We have multiple images in a known relation and available reference appearance (i. e., light, materials). Our approach relaxes those two requirements: we use a discriminator that can work on arbitrary projections and arbitrary natural input images, without known reference.

3D Shape From 2D Photo Collections
We now introduce PLATONICGAN (Fig. 2). The rendering layers used here will be introduced in Sec. 4.
Common GAN Our method is a classic (generative) adversarial design [14] with two main differences: The discriminator D operates in 2D while the 3D generator G produces 3D output. The two are linked by a fixed-function projection operator, i. e., non-learnable (see Sec. 4).
Let us recall the classic adversarial learning of 3D shapes [36], which is a min-max game between the discriminator and the generator cost, respectively c Dis and c Gen . The discriminator cost is where D Ψ is the discriminator with learned parameters Ψ which is presented with samples x from the distribution of real 3D shapes x ∼ p Data . Here E p denotes the expected value of the distribution p.
The generator cost is where G Θ is the generator with parameters Θ that maps the latent code z ∼ p Gen to the data domain.
PLATONICGAN The discriminator cost is calculated identical to the common GAN with the only difference that the input samples are rendered 2D images with generation cost where R projects the generator result G Θ (z) from 3D to 2D along the sampled view direction ω. See Sec. 3.1 for details. While many parameterizations for views are possible, we choose an orthographic camera with fixed upright orientation that points at the origin from an Euclidean position ω ∈ S 2 on the unit sphere. E pView(ω) is the expected value across the distributions ω ∼ p View of views.

PLATONICGAN 3D Reconstruction
Two components in addition to our Platonic concept are required to allow for 3D reconstruction, resulting in where c Gen includes an encoding step and c Rec encourages the encoded generated-and-projected result to be similar to the encoder input where λ = 100. We detail both of these steps in the following paragraphs: Generator The generator G Θ does not directly work on a latent code z, but allows for an encoder E Φ with parameters Φ that encodes a 2D input image I to a latent code z = E Φ (I). The cost becomes, Reconstruction We encourage the encoder E Φ and generator G Θ to reproduce the input in the L 2 sense: by convention the input view is ω 0 = (0, 0), where y represents the ground truth image. While this step is not required for generation it is mandatory for reconstruction. Furthermore, it adds stability to the optimization as it is easy to find an initial solution that matches this 2D cost before refining the 3D structure.

Optimization
Two key properties are essential to successfully optimize our PLATONICGAN: First, maximizing the expected value across the distribution of views p View and second, back-propagation through the projection operator R. We extend the classic GAN optimization procedure in Alg. 1.

Algorithm 1 PLATONICGAN Reconstruction Update
Step Projection We focus on the case of a 3D generator on a regular voxel grid v nc×n 3 p and a 2D discriminator on a regular image I nc×n 2 p where n c denotes the number of channels and n p = 64 corresponds to the resolution. In section 4, we discuss three different projection operators. We use R(ω, v) to map a 3D voxel grid v under a view direction ω ∈ S 2 to a 2D image I.
We further define R(ω, v) := ρ(T(ω)v) with rotation matrix T(ω) according to the view direction ω and an image formation function ρ(v) that is view-independent. The same transformation is shared by all implementations of the rendering layer, so we will only discuss the key differences of ρ in the following. Note that a rotation and a linear resampling is back-propagatable and typically provided in a deep learning framework, e. g., as torch.nn.functional.grid sample in PyTorch [23]. While we work in orthographic space, ρ could also model a perspective transformation.
View sampling We assume uniform view sampling.

Rendering Layers
Rendering layers (Fig. 3) map 3D information to 2D images so they can be presented to a discriminator. We first assume the 3D volume to be rotated (Fig. 3, a) into camera space from view direction ω (Fig. 3, b), such that the pixel value p is to be computed from all voxel values v i and only those (Fig. 3, c). The rendering layer maps a sequence of n z voxels to a pixel value ρ(v) ∈ R nc×n 3 p → R nc×n 2 p . Composing the full image I just amounts to executing ρ for every pixel p resp. all voxels v = v 1 , . . . , v nz at that pixel.
Note, that the rendering layer does not have any learnable parameters. We will now discuss several variants of ρ, implementing different forms of volume rendering [7].   shows the image formation models we currently support. Visual hull (VH) Visual hull [21] is the simplest variant (Fig. 4). It converts scalar density voxels into binary opacity images. A voxel value of 0 means empty space and a value of 1 means fully occupied, i. e., v i ∈ [0, 1]. Output is a binary value indicating if any voxel blocked the ray. It is approximated as Note that the sum operator can both be back-propagated and is efficiently computable on a GPU using a parallel scan. We can apply this to learn 3D structure from binary 2D data such as segmented 2D images.

Absorption-only (AO)
The absorption-only model is the gradual variant of visual hull. This allows for "softer" attenuation of rays. It is designed as: If v i are fractional the result is similar to an x-ray, i. e., v i ∈ [0, 1]. This image formation allows learning from xrays or other transparent 2D images. Typically, these are single-channel images, but a colored variant (e. g., x-ray at different wavelength or RGB images of colored transparent objects) could technically be done.
Emission-absorption (EA) Emission-absorption allows the voxels not only to absorb light coming towards the observer but also to emit new light at any position. This interplay of emission and absorption can model occlusion, which we will see is useful to make 3D sense of a 3D world. Fig. 3 uses emission-absorption with high absorption, effectively realizing an opaque surface with visibility. A typical choice is to have the absorption v a monochromatic and the emission v e chromatic.

The complete emission-absorption equation is
While such equations are typically solved using raymarching [7], they can be rewritten to become differentiable in practice: First, we note that the transmission t i from voxel i is one minus a product of one minus the density of all voxels before i. Similar to a sum such a cumulative product can be back-propagated and computed efficiently using parallel scans, e. g., using torch.cumprod. A numerical alternative, that performed similar in our experiments, is to work in the log domain and use torch.cumsum.

Evaluation
Our evaluation comprises of a quantitative (Sec. 5.4) and a qualitative analysis (Sec. 5.5) that compares different previous techniques and ablations to our work (Sec. 5.2).

Data sets
Synthetic We evaluate on two synthetic data sets: (a) ShapeNet [5] and (b) mammalian skulls [16]. For our quantitative analysis, we use ShapeNet models as 3D ground truth is required, but strictly only for evaluation, never in our training. 2D images of 3D shapes are rendered for the three image formation models VH, AO, EA. Each shape is rendered from a random view (50 per object), with random natural illumination. ShapeNet only provides 3D density volumes which is not sufficient for EA analysis. To this end, we use volumetric projective texturing to propagate the appearance information from thin 3D surface crust as defined by ShapeNet's textures into the 3D voxelization in order to retrieve RGBA volumes where A corresponds to density. We use shapes from the classes airplane, car, chair, rifle and lamp. The same train / validation / test split as proposed by [5] is adopted.
We also train on a synthetic x-ray data set that consists of 466,200 mammalian skull x-rays [16]. We used the monkey skulls subset of that data set (∼30k x-rays).

Real
We use two data sets of rare classes: (a) chanterelle (60 images) and (b) tree (37 images) (not strictly rare, but difficult to 3D-model). These images are RGBA, masked, on white background. Note, that results on these input data has to remain qualitative, as we lack the 3D information to compare to and do not even have a second view of the same object to even perform an image comparison.

Baselines and comparison
2D supervision First, we compare the publicly available implementation of PrGAN [11] with our Platonic method. PrGAN is trained on an explicitly created data set adhering to their view restrictions (8 views along a single axis). Compared to our method it is only trained on visual hull images, however for evaluation purposes absorption-only and emission-absorption (in form of luminance) images are used as input images at test time. Note that PrGAN allows for object-space view reconstruction due to view information in the latent space whereas our method performs reconstruction in view-space. Due to the possible ambiguities in the input images (multiple images can belong to the same 3D volume), the optimal transformation into object space is found using a grid search across all rotations.
3D supervision The first baseline with 3D supervision is MULTI-VIEW, that has training-time access to multiple images of the same object [38] in a known spatial relation. Note, that this is a stronger requirement than for PLATONIC-GAN that does not require any structure in the adversarial examples: geometry, view, light -all change, while in this method only the view changes in a prescribed way.
The second competitor is a classic 3DGAN [36] trained with a Wasserstein loss [2] and gradient penalty [15].
To compare PLATONICGAN against methods having access to 3D information, we also propose a variant PLA-TONIC3D by adding the PLATONICGAN adversarial loss term (for all images and shapes) to the 3DGAN framework.

Evaluation Metrics
2D evaluation measures Since lifting 2D information to 3D can be ambiguous, absolute 3D measures might not be the best suitable measures for evaluation on our task. For instance, a shift in depth of an object under an orthographic camera assumption will result in a higher error for metrics in 3D, but the shift would not have any effect on a rendered image. Thus, we render both the reconstructed and the reference volume from the same 10 random views and compare their images using SSIM / DSSIM [33] and VGG16 [27] features. For this re-rendering, we further employ four different rendering methods: the original (i. e., ρ) image formation (IF), volume rendering (VOL), iso-surface rendering with an iso-value of .1 (ISO) and a voxel rendering (VOX), all under random natural illumination.

3D evaluation measures
We report root-mean-squarederror (RMSE), intersection-over-union (IoU) and chamfer distance (CD). For the chamfer distance we compute a weighted directional distance:  where T and O correspond to output and target volumes respectively, and w j denotes the density value of the voxel at location p j . The weighting makes intuitive sense as our results have scalar values rather than binary values, i. e., higher densities get penalized more, and N is the total number of voxels in the volume. We give preference to such a weighting opposed to finding a threshold value for binarization.

Quantitative evaluation
Tbl. 2 summarizes our main results for the airplane class. Concerning the image formation models, we see that the overall values are best for AO, which is expected: VH asks for scalar density but has only a binary image; AO provides internal structures but only needs to produce scalar density; EA is hardest, as it needs to resolve both density and color. Nonetheless the differences between us and competitors are similar across the image formation models.
2D supervision We see that overall, our 2D supervised method outperforms PrGAN for VH and AO. Even though PrGAN was not trained on EA it wins for all metrics against our 2D supervised method. However, it even outperforms the 3D supervised methods 3DGAN and MULTI-VIEW which demonstrates the complexity of the task itself. However, PrGAN for EA only produces density volumes unlike all other methods that produce RGBA volumes. Comparing our 2D supervised method against the 3D supervised methods we see that overall our method produces competitive results. Regarding MULTI-VIEW we sometimes even perform better.
3D supervision Comparing our PLATONIC3D variant to the 3D baselines we observe our method to mostly outperform them for 2D metrics. Not surprisingly our method performs worse for 3D metrics as our approach only operates in 2D.
In Tbl. 3 we look into the performance across different classes. rifle performs best: the approach learns quickly from 2D that a gun has an outer 3D shape that is a revolute structure. chair performs worst, likely due to its high intra-class variation. In Tbl. 4 we compare the mean VGG error of a vanilla 3D GAN trained only on 3D shapes, a Platonic approach accessing only 2D images, and PLATONIC3D that has ac-  cess to both. We keep the number of 2D images fixed, and increase the number of 3D shapes available; the horizontal axis in Tbl. 4. Without making use of the 3D supervision, the error of PLATONICGAN remains constant, independent of the number of 3D models. Like this, we see that a PLA-TONICGAN (red line) can beat both other approaches in a condition where little 3D data is available (left). When more 3D data is available, PLATONICGAN (green line) wins over a pure 3D GAN (blue line). We conclude that adding 2D image information to a 3D corpus helps, and when the corpus is small enough even performs better than 3D-only supervised methods.

Qualitative
Synthetic Fig. 5 shows typical results for the reconstruction task. We see that our reconstruction can produce airplane, chair and rifle 3D models representative of the input 2D image. Most importantly, these 3D models look plausible for multiple views, not only from the input one. The results on the chair category also show that the model captures the relevant variation, ranging from straight chairs over club chairs to armchairs. For gun, the results turn out almost perfect, in agreement with the numbers reported before. In summary, our quality is comparable to GANs with 3D supervision.
Synthetic rare We explored reconstructing skulls from xray (i. e., the AO IF model) images [16] in Fig. 9. We find the method to recover both external and internal structures.
Real rare Results for rare classes are seen in Fig. 1 and Fig.  Fig. 7. We see that our method produces plausible details from multiple views while respecting the input image, even in this difficult case. No metric can be applied to these data as no 3D volume is available to compare in 3D or re-project.

Discussion
Why not having a multi-view discriminator? It is tempting to suggest a discriminator that does not only look at a single image, but at multiple views at the same time to judge if the generator result is plausible holistically. But while we can generate "fake" images from multiple views p Data , the set of "real" natural images does not come in such a form. As a key advantage, our method only expects unstructured data: online repositories hold images with unknown camera, 3D geometry or illumination.
Failure cases are depicted in Fig. 8. Our method struggles to reconstruct the correct pose as lifting 2D images to 3D shapes is ambiguous for view-space reconstruction. Figure 6. Comparison of 3D reconstruction results using the class plane between different forms of supervision (columns) for two different input views (rows). PLATONICGAN, in the second column, can reconstruct a plausible plane, but with errors such as a wrong number of engines. The 3D GAN in the third column fixes this error, but at the expense of slight mode collapse where instances look similar and slightly "fat". Combining a 3D GAN with adversarial rendering as in the fourth row, is closest to the reference in the fifth row. Figure 7. 3D Reconstruction of different trees using the emissionabsorption image formation model, seen from different views (columns). The small images were used as input. We see that PLATONICGAN has understood the 3D structure, including a distinctly colored stem, fractal geometry and structured leave textures. The encoder is unable to estimate the correct camera-pose due to view-ambiguities in the input image and symmetries in the shapes. The generator then tries to satisfy multiple different camera-poses.

Conclusion
In this paper, we have presented PLATONICGAN, a new approach to learning 3D shapes from unstructured collections of 2D images. The key to our "escape plan" is to train a 3D generator outside the cave that will fool a discriminator a ) 2 D I n p u t X -r a y b ) X -r a y c ) V o l u me r e n d e r Figure 9. PlatonicGANs trained on 2D x-rays (i. e., AO IF) of mammalian skulls (a). The resulting 3D volumes can be rendered from novel views using x-ray (b) and under novel views in different appearance, here, using image-based lighting (c).
seeing projections inside the cave. We have shown a family of rendering operators that can be GPU-efficiently back-propagated and account for occlusion and color. These support a range of input modalities, ranging from binary masks, over opacity maps to RGB images with transparency. Our 3D reconstruction application is build on top of this idea to capture varied and detailed 3D shapes, including color, from 2D images. Training is exclusively performed on 2D images, enabling 2D photo collections to contribute to generating 3D shapes.
Future work could include shading that is related to gradients of density [7] into classic volume rendering. Furthermore, any sort of differentiable rendering operator ρ can be added. Devising such operators is a key future challenge. Other adversarial applications such as 2D supervised completion of 3D shapes seems worth exploring. Enabling object-space as opposed to view-space reconstruction would help to prevent failure cases as shown in Fig. 8.
While we combine 2D observations with 3D interpretations, similar relations might exist in higher dimensions, between 3D observations and 4D (3D shapes in motion) but also in lower dimensions, such as for 1D row scanner in robotics or 2D slices of 3D data such as in tomography.