Accelerating 3D Deep Learning with PyTorch3D
  Deep learning has significantly improved 2D image recognition. Extending into
3D may advance many new applications including autonomous vehicles, virtual and
augmented reality, authoring 3D content, and even improving 2D recognition.
However despite growing interest, 3D deep learning remains relatively
underexplored. We believe that some of this disparity is due to the engineering
challenges involved in 3D deep learning, such as efficiently processing
heterogeneous data and reframing graphics operations to be differentiable. We
address these challenges by introducing PyTorch3D, a library of modular,
efficient, and differentiable operators for 3D deep learning. It includes a
fast, modular differentiable renderer for meshes and point clouds, enabling
analysis-by-synthesis approaches. Compared with other differentiable renderers,
PyTorch3D is more modular and efficient, allowing users to more easily extend
it while also gracefully scaling to large meshes and images. We compare the
PyTorch3D operators and renderer with other implementations and demonstrate
significant speed and memory improvements. We also use PyTorch3D to improve the
state-of-the-art for unsupervised 3D mesh and point cloud prediction from 2D
images on ShapeNet. PyTorch3D is open-source and we hope it will help
accelerate research in 3D deep learning.
