Neural Puppet: Generative Layered Cartoon Characters
  We propose a learning based method for generating new animations of a cartoon
character given a few example images. Our method is designed to learn from a
traditionally animated sequence, where each frame is drawn by an artist, and
thus the input images lack any common structure, correspondences, or labels. We
express pose changes as a deformation of a layered 2.5D template mesh, and
devise a novel architecture that learns to predict mesh deformations matching
the template to a target image. This enables us to extract a common
low-dimensional structure from a diverse set of character poses. We combine
recent advances in differentiable rendering as well as mesh-aware models to
successfully align common template even if only a few character images are
available during training. In addition to coarse poses, character appearance
also varies due to shading, out-of-plane motions, and artistic effects. We
capture these subtle changes by applying an image translation network to refine
the mesh rendering, providing an end-to-end model to generate new animations of
a character with high visual quality. We demonstrate that our generative model
can be used to synthesize in-between frames and to create data-driven
deformation. Our template fitting procedure outperforms state-of-the-art
generic techniques for detecting image correspondences.
