Music-to-Dance Generation with Optimal Transport
  Dance choreography for a piece of music is a challenging task, having to be
creative in presenting distinctive stylistic dance elements while taking into
account the musical theme and rhythm. It has been tackled by different
approaches such as similarity retrieval, sequence-to-sequence modeling and
generative adversarial networks, but their generated dance sequences are often
short of motion realism, diversity and music consistency. In this paper, we
propose a Music-to-Dance with Optimal Transport Network (MDOT-Net) for learning
to generate 3D dance choreographies from music. We introduce an optimal
transport distance for evaluating the authenticity of the generated dance
distribution and a Gromov-Wasserstein distance to measure the correspondence
between the dance distribution and the input music. This gives a well defined
and non-divergent training objective that mitigates the limitation of standard
GAN training which is frequently plagued with instability and divergent
generator loss issues. Extensive experiments demonstrate that our MDOT-Net can
synthesize realistic and diverse dances which achieve an organic unity with the
input music, reflecting the shared intentionality and matching the rhythmic
articulation. Sample results are found at
https://www.youtube.com/watch?v=dErfBkrlUO8.
