{'fraction_non_alphanumeric': 0.028074399099134083, 'fraction_numerical': 0.011920941249563157, 'mean_word_length': 4.9177389705882355, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
SurfGen: Adversarial 3D Shape Synthesis with Explicit Surface Discriminators

Recent advances in deep generative models have led to immense progress in 3D shape synthesis. While existing models are able to synthesize shapes represented as voxels, point-clouds, or implicit functions, these methods only indirectly enforce the plausibility of the final 3D shape surface. Here we present a 3D shape synthesis framework (SurfGen) that directly applies adversarial training to the object surface. Our approach uses a differentiable spherical projection layer to capture and represent the explicit zero isosurface of an implicit 3D generator as functions defined on the unit sphere. By processing the spherical representation of 3D object surfaces with a spherical CNN in an adversarial setting, our generator can better learn the statistics of natural shape surfaces. We evaluate our model on large-scale shape datasets, and demonstrate that the end-to-end trained model is capable of generating high fidelity 3D shapes with diverse topology.


Introduction
Shape generation is primarily concerned with the synthesis of diverse, realistic, and novel shapes. High fidelity models of 3D shapes are key to creating immersive virtual worlds, and are important to many disciplines, including architecture, visual effects, and training robots in simulated navigation.
With the introduction of large scale 3D object datasets such as ShapeNet [6] and ModelNet [62], there has been significant progress towards building generative models of 3D shapes. The predominant approach is to perform training using representations that are straightforward for neural networks to work with, such as voxel grids, point clouds, or implicit functions. This is in contrast to the majority of graphics & simulation tasks, which require an triangle mesh representation of a shape. When learned in an adversarial setting, the discriminator in these frameworks only indirectly ensure the realism of the object surface.
A major barrier in training a neural network on object surfaces is the inherent irregularity and discrete nature of 3D surfaces. Different objects within the same object class can have vastly different topologies, and can be characterized by different internal connectivity when represented as a triangle mesh. Our key insight is that a discriminator ap- plied to the surface should focus on the geometric properties of the surface (curvature, orientation, topology, etc.), while ignoring properties that are irrelevant to the shape. Here we propose to transform object surface meshes to spherical maps computed with a differentiable function f M→S . A spherical map is a surface representation defined on discrete samples from the surface of a sphere. The values at each pixel represents a minimal distance of the object surface along a ray, as well as the surface occupancy along the ray. The spherical map is appealing because it is a singular representation that captures the surface geometry of a shape. Given a spherical map, we then utilize a network with spherical convolution layers to complete our discriminator.
We propose SurfGen, an end-to-end generative model of 3D shapes that is trained with a discriminator which operates on the explicit zero isosurface of an implicit shape function. We demonstrate that applying adversarial training on the surface of an object leads to generated highly realistic shapes. This results in an approach that can generate high quality shapes with arbitrary topology and resolution. Examples of shapes generated by our model are shown in Figure 1. In summary our contributions are three fold: • We introduce a spherical projection operator that takes as input an explicit triangle mesh, and is fully differentiable w.r.t. the vertices. • We propose SurfGen, an end-to-end differentiable 3D shape synthesis framework which applies an adversarial objective on the zero isosurface of the generator. • We demonstrate our model can synthesize realistic, high quality shapes that have diverse topology.  Figure 2. Components of our model. Top: During test time, we sample a random code and extract the triangle mesh; Bottom: During training, we extract the surface of the shape represented as a triangle mesh, and compute the differentiable spherical projection of the surface. The spherical surface map is fed to a spherical discriminator. Note that we do not utilize an encoder during training.

Related Work
Our method is related to prior work on learning statistical models for 3D shape analysis and generation. In this section we will discuss deep learning based models for 3D shapes, spherical projections, and differentiable rendering.
Deep learning for 3D shapes. Recent advances in deep learning have enabled models capable of image-guided shape reconstruction and novel shape synthesis. 3D-R2N2 [9] proposed learning a recurrent model for voxel based reconstruction with multiple views, [16,19,60,69] further improved image based voxel reconstruction. Due to memory constraints, these methods are usually limited in terms of their resolution. Methods have also been proposed to generate point clouds from images [14,56], however the unordered nature of point clouds limits the fidelity of the final 3D reconstructions. Some approaches have been proposed to deform a mesh template [23,57,49], or a set of surface patches [18]. These techniques are typically limited to modeling shapes of fixed topology, or produce meshes that require post-processing to become usable. It has become possible recently to regress implicit functions in the form of binary occupancy or signed distances [63,8,38,41]. Similarly, other forms of learned 3D priors have been applied to convert or refine existing shape representations [59,4,13].
Generative models for 3D shapes. Modern generative models for 3D shapes generally utilize an adversarial constraint, take a variational approach, or use flow-based models. Of note, [62] design a deep belief network to synthesize novel shapes when trained on a large-scale dataset. [61] proposed using a generative adversarial network (GAN) for voxels. [1,68] proposes learning latent-GANs [37] for point clouds, while [28] proposes an autoencoding objective to stabilize the training of point cloud generative models. Similarly [15,53,46] advocate for modified generators to improve point cloud generation. Implicit generative models based on latent-GANs [8], and point cloud discriminators [27] have also been proposed. While these methods are capable of generating high resolution shapes, they still struggle with thin structures and implausible objects.
Spherical representations. A spherical representation captures information about a shape on the surface of a sphere. For the purpose of shape retrieval, [2,25,54] represent shapes as spherical distance functions, [66] captures the number of surface intersections. Spherical representations have also been used in modern deep learning systems for object recognition [5]. [69] uses a non-differentiable spherical representation computed from voxel grids to facilitate 3D shape reconstruction in the projected spherical space, demonstrating the effectiveness of spherical projections for shape synthesis. In our work, we propose a fully differentiable spherical projection to capture the statistics of surfaces, allowing for the end-to-end training of our generator.
Differentiable rendering. A spherical projection can be viewed as a rendering operation using a non-linear projection operator. Because the rendering operation is normally discrete, it does not provide usable error gradients for optimization. A variety of mesh [10,33,29,24,31,8,43], point-cloud, and implicit [32,40,39,48] based differentiable renderers have been proposed. [24] developed an approximation of gradient for rasterization. [50] proposes learning high frequency faces details using a differentiable renderer. [35] also demonstrated the feasibility of differentiable rendering in 3D scene optimization. [64] proposed a differentiable volume sampling method which can approximate the ray tracing algorithm. Other works such as [22,21] also implemented differentiable operation for single image 3D shape reconstruction tasks. Our work enables gradient based optimization in a spherical projection layer, allowing a generator to be optimized by error signals defined in the spherical domain.

Approach
We propose SurfGen, an end-to-end fully differentiable framework for the generation of 3D shapes. Key to our model, is a differentiable spherical projection operator which allows surfaces to be represented as a spherical projection map, where an adversarial loss can be naturally applied. Our model is illustrated in Figure 2.

3D Shape Generator
We adopt DeepSDF [41] as the generator in our model. In this generator, each shape is represented by an implicit signed distance function (SDF). For each point p = (p x , p y , p z ) and a given shape s, the SDF encodes the distance of the point to the nearest surface: SDF s (p) = d, d ∈ R. The sign of d encodes if the point is inside (negative) or outside (outside) a given shape. Our generator g ψ is trained to map a randomly sampled latent code z and a position p to a corresponding SDF value: The surface of a shape is implicitly represented by the zero isosurface of the SDF.

Differentiable triangle mesh extraction
In order to apply a discriminator to the surface of an object represented as an implicit SDF, we need to first find the zero isosurface. We choose to utilize marching cubes [34] to extract an explicit triangle mesh from the signed distance function. While sphere tracing can also be utilized to extract the zero isosurface from a signed distance function, we found the required speed-accuracy trade-off unsuitable for use in training.
Instead, we use the MeshSDF method proposed in [44] for differentiable isosurface extraction. During training, we evaluate our signed distance function generator g ψ on an euclidean grid of size 128 3 in the range of [−1, 1], and use marching cubes (MC) to extract the surface as a triangle is the set of mesh vertices represented in R 3 , and F represents the set of triangle faces enclosed by the edges.
We use the loss from a discriminator D φ to compute a gradient with respect to the vertices V from the zero isosurface. For a discriminator loss L differentiable w.r.t. vertices V , the gradient with respect to generator weights ψ can be computed by evaluating: This approach to differentiable surface extraction allows discriminator gradients present on the vertices V to modify surface shape and topology by changing the underlying signed distance function. We refer readers to [44] for a proof and discussion of this method.

Differentiable spherical surface projection
In this section, we present our diffentiable spherical projection layer. Consider a 3D object parameterized as a triangle mesh M = (V, F ). Because surfaces themselves can vary in topology, and meshes can themselves vary in internal connectivity, it is non-trivial to train a neural network on the surface on an object.
Our spherical projection layer f M→S transforms an irregular 3D mesh into a regular spherical domain. The support of the spherical representation is defined as discrete samples on the unit sphere There are two major challenges to adopting such a framework. First, the pixel coordinates lie on the surface of a sphere parameterized by (θ, φ), which prevents the use of a projection matrix to express the vertex transformations. Second, the rasterization and z-buffer algorithm are discrete operations, which causes discontinuities in the back-propagated gradients as triangles change in depth or move laterally. We use a similar approach to those presented in [31,7] to enable a differentiable spherical surface projection function.
Because of the non-linear projection, we utilize raycasting to find intersections of the object surface with rays that originate from the unit sphere. Each ray R i is defined as a six tuple representing origin and direction in euclidean coordinates: We modify the ray-intersection kernel to output direct intersections, as well as "near misses" where the ray is within distance r of a triangle. The ray-intersection kernel is repeated such that the k nearest hits along each ray are retrieved: At an intersection location − → p j i inside triangle j for R i , we compute a pixel attribute u i j as an barycentric interpolation of vertex attributes   [67], ShapeGAN w/ voxel [27] discriminator, ShapeGAN w/ PointNet discriminator [27], and DeepSDF with a latent generator [1,8].
where the barycentric weights are computed using the differentiable function Ω. When computing a spherical depth projection, we set the attributes to be the (x, y, z) coordinate of each vertex in triangle j: Beyond the spherical depth map, the spherical silhouette of a shape can also be computed such that it is differentiable. We utilize "near miss" rays that do not intersect inside a triangle, but has a point on that ray p j i that is within distance r outside of a triangle. We compute the squared euclidean distance between the ray and the closest point on the triangle: Where δ is a hyper-parameter that controls how fast the influence decays as a function of distance. We use a product based function to aggregate all the near miss collisions for a given ray: A ray is illustrated in Figure 4. This allows for the efficient computation of spherical depth and occupancy maps in a way that is fully differentiable with respect to the vertex attributes.

Choice of sampling and projection
Sampling on a Sphere. A discretization of the sphere needs to be selected prior to spherical projection. Common sampling schemes includes Driscoll and Healy [12], icosahedral [58], and HEALPix [17]; which use rectangular, icosahedral, and equal area sampling respectively. Due to our efficient ray-casting based implementation, the differentiable spherical projection layer can work with any discretization of the sphere. In practice, the HEALPix based sampling with 12, 288 points is used due to the balanced area assigned to each point on the sphere.
Choice of Projection. Our spherical projection layer can model arbitrary projections. The most simple projection operator is the radial spherical projection, which gives a ray direction to be (D where O is discrete sample on the sphere. It was shown in [69], that a radial projection may have poor coverage of a shape due to self-occlusions. They address this by combining the spherical representation with an additional non-radial projection. Taking inspiration from orthographic map projections, we propose orthographic spherical projection where (D x , D y , D z ) = −(0, 0, O z ), where z is assumed to be the gravity aligned (top-down) axis. Our full spherical projection combines the spherical depth map from radial & orthographic spherical projections, and use the silhouette map from the orthographic projection. The radial silhouette map is omitted since most radial rays have direct intersections within a triangle face, and do not provide a useful training signal. We show how the projections compare in Figure 5.

Discriminator Implementation
Network Architecture. As our spherical maps are discrete samples on a unit sphere and non-euclidean, we cannot use a regular 2D or 3D convolution. We implement our discriminator using the graph-based spherical convolution layers proposed by [11]. Our discriminator takes as input the 3 spherical maps described in section 3.4. The network consists of 5 residual spherical blocks that each perform average pooling to reduce the total number of spherical samples by a factor of 4, as well as a single self-attention layer in the discriminator. We average pool the final 12 pixels, 256 channel spherical maps, and use fully connected layers to produce the scalar discriminator output. We utilize Instance Normalization [52] and Leaky ReLU [36] in the discriminator.

Overall model
The straightfoward approach to training an implicit generator with a surface discriminator would be to update the implicit generator and surface discriminator in an alternate fashion. However, unlike the SDF which is defined for all p ∈ R 3 , the surface only exists at select locations. We need to constrain the generator such that the location of the isosurface is within the spherical projection layer, and ideally approximate to the desired shape. Methods using geometric initializations [3], meta-learning [47], and variational methods [20] have been proposed to stabilize training and accelerate convergence. We choose to regularize our generator with a variational autodecoder (VAD) [20] loss alongside our adversarial criterion. This VAD-GAN setup preserves the generator+discriminator structure of a generative adversarial network (GAN) during training, only requiring the addition of a light-weight embedding layer to the model, while significantly outperforming the original VAD only objective.
During training, we sample a latent code z from the approximate posterior for a given shape x j modeled as a multivariate Gaussian with diagonal covariance: We use the reparameterization trick [26] to allow for direct optimization of latent parameters µ j , σ j , and set z j = µ j +σ j , where ∼ N (0, I). During training, the generator and latent parameters are trained to minimize the the adversarial objective L GAN and maximize the evidence lower bound of the marginal likelihood (ELBO): Top Bottom Initialization Target optimization steps Figure 6. We optimize a MSE loss on the spherical representation between the target and initial shape. From top to bottom: the top view of the radial projection, the bottom view of the radial projection, rendered SDF output. This shows that our spherical surface projection can induce topology and shape changes.
Equation 13 could be approximated by sampling from 3D space following a certain distribution defined in [41]. For the final results, we use clamped version of L1 loss for L SDF (·). To stabilize GAN training, we employ standard hinge GAN loss L hinge [30,51] with a small amount of feature matching loss L feat [45] applied to the first four discriminator feature maps. In summary, the overall objective is summarized in Equation 15 where we use α = 1, β = 1 and γ = 1e −5 , λ = 0.5.
In order to facilitate stable training, we zero the gradients in the generated depth maps where there is no surface occupancy in the corresponding target shape: To ensure that surface gradients w.r.t. the SDF generator are within the same magnitude as the other losses, we scale the gradient from Equation 3 with a constant ω = 1e −4 .
Our model is implemented using Pytorch, with the ray casting implemented in embree [55,42], and marching cubes implemented in CUDA [65].

Validating end-to-end optimization
We first verify the differentiability of our spherical projection layer. Using the data from [44], we train a generator network g ψ to approximate the signed distance function of two shapes: a genus zero cow, and a genus one rubber duck.  Table 1. We compare the effectiveness of four spherical features in terms of Chamfer and earth movers distance (EMD) during the optimization process. Combined indicates the radial depth, ortho depth, and ortho silhouette are used together. Bold indicates best method at each iteration.
The generator learns to associate a latent code with a corresponding implicit distance field. The loss during optimization is the pixel-wise mean squared error (MSE) between the current and target spherical projections: The gradients are backpropagated and are used to optimize the latent code z. As shown in Figure 6, our spherical projection layer can modify the underlying implicit representation, and can change both the shape and topology of an object. We quantitatively compare the effect of using radial depth, radial silhouette, orthographic depth, orthographic silhouette, as well as combined (radial depth + orthographic depth + orthographic silhouette) as features for our optimization in Table 1. Because few rays in the radial projection experience a "near miss" for the shape, the radial silhouette does not provide useful gradients for the optimization process. The loss that uses the three combined features provides the fastest convergence when measured by chamfer distance or earth movers distance. The final projection used in our shape generation experiment uses the combined features.

Shape Generation
Data preparation. All models are trained on one of three categories from ShapeNet.v2 [6]: airplane, car, and chair.
To ensure comparability, we use the official training split. Each shape is centered and normalized to the unit sphere. We utilize the improved signed distance generation method proposed by the authors of ShapeGAN. Each shape is rendered from 50 equidistance views, the depth buffer is projected into object space to compute the surface point cloud. A point is considered to be outside the shape if it is seen by any camera. Shapes are discarded if fewer than 0.5% of the points are inside. We apply a small negative offset (2e −3 ) to SDF values during training to facilitate surface extraction. This results in 2788, 2452, 4550 shapes in the training set for the airplane, car, and chair categories respectively. We use the same sampling scheme used in DeepSDF. For the ground truth test set, we uniformly query points in a unit sphere, and randomly select 2048 points thresholded to lie near the surface.
Baselines. We compare against four alternative models for implicit shape generation: two variants of Shape-GAN [27] which use a voxel and PointNet as discriminator, a latent-GAN trained on DeepSDF embeddings, and a VAD based SDF model which does not utilize our surface discriminator. We use the VAD layer implementation provided in [20]. For the latent-GAN, VAD, and our SurfGen model, we use the DeepSDF network as backbone. For ShapeGAN the author provided code and hyperparameters are used.
Evaluation and metrics. Every mesh is extracted using marching cubes at 256 3 resolution, and 2048 points are randomly sampled from each mesh. The results are compared using the metrics introduced by [1]. Minimum matching distance (MMD) measures the distance of a shape from the test set to its nearest neighbor, and can be understood to be a proxy for fidelity. Coverage (COV) measures the fraction of the test set that are the nearest neighbor to a sample in the generated set, and can be understood to be a proxy for diversity. For each measure, the distance metric can either be chamfer distance (CD) or earth movers distance (EMD).
Results. Qualitative results for shapes generated by each method are shown in Figure 3 and Figure 7, while quantitative results are shown in Table 2. Shapes generated by our model have high-quality surfaces that are largely free from the high frequency artifacts present in Latent-GAN, and the block like artifacts from the two ShapeGAN variants. We also observe many thin shell like artifacts on VAD synthesized airplanes. For the chair class, the VAD only method SurfGen has the lowest MMD value across all evaluated classes for both CD & EMD, this indicates that our methods synthesizes shapes are closely matched to samples from the test set. SurfGen produces high coverage for the airplane class, while it is tied for coverage in the chair class. Our method is less competitive in the car class. A possible reason is the lack of high frequency geometry normally present on the outside surface of cars. This causes cars to be difficult to learn using a surface based adversarial loss.

Latent Analysis
Individual latent dimensions. We explore if individual dimensions in our latent space have semantic meaning. Given a randomly sampled latent code, we select a dimension and increase its value in small increments. Our results are shown in Figure 8. We observe that changing certain dimensions can induce structural changes in a shape.
Interpolating between latents. In Figure 9, we demonstrate that shapes can smoothly change as we linearly interpolate between two randomly sampled latent vectors, even between shapes that have different topologies.
Computational efficiency. For a batch of 8 shapes, on a dual Nvidia 3090 system the grid query step at 128 3 resolution takes approximately 2, 500 ms, while marching cubes takes 90 ms. The spherical projection takes around 600 ms on a 12 core CPU. For G and D combined, the for-  Figure 9. Visualization of the shape as we linearly interpolate between two randomly sampled latent vectors. ward and backwards pass for a single batch takes around 5 seconds total.

Failure Cases
Certain shapes do not have well behaved spherical projections. In the most common case, we observe that this is mostly due to a shape having protrusions along the gravity aligned axis that significantly affect the centering of a shape into a unit sphere. The issue is most prominent in airplanes, while this is to a large degree mitigated by using the orthographic projection, we observe rare ring-like artifacts in airplanes as shown in Figure 10 that seem to be induced by the spherical discriminator. As part of future work, it may make sense to investigate the integration of the surface discriminator with discriminators directly applied to the implicit field.

Conclusion
In this paper, we have introduced a novel shape synthesis method that allows a discriminator to focus directly on the surface on a generated shape. Our method is capable of generating diverse high quality shapes with complex geometry. Our model has diverse downstream applications as part of a larger 3D synthesis pipeline. We hope our work will inspire future research in 3D shape synthesis.