Cycle-Consistent Generative Rendering for 2D-3D Modality Translation
  For humans, visual understanding is inherently generative: given a 3D shape,
we can postulate how it would look in the world; given a 2D image, we can infer
the 3D structure that likely gave rise to it. We can thus translate between the
2D visual and 3D structural modalities of a given object. In the context of
computer vision, this corresponds to a learnable module that serves two
purposes: (i) generate a realistic rendering of a 3D object (shape-to-image
translation) and (ii) infer a realistic 3D shape from an image (image-to-shape
translation). In this paper, we learn such a module while being conscious of
the difficulties in obtaining large paired 2D-3D datasets. By leveraging
generative domain translation methods, we are able to define a learning
algorithm that requires only weak supervision, with unpaired data. The
resulting model is not only able to perform 3D shape, pose, and texture
inference from 2D images, but can also generate novel textured 3D shapes and
renders, similar to a graphics pipeline. More specifically, our method (i)
infers an explicit 3D mesh representation, (ii) utilizes example shapes to
regularize inference, (iii) requires only an image mask (no keypoints or camera
extrinsics), and (iv) has generative capabilities. While prior work explores
subsets of these properties, their combination is novel. We demonstrate the
utility of our learned representation, as well as its performance on image
generation and unpaired 3D shape inference tasks.
