AudioViewer: Learning to Visualize Sounds
  A long-standing goal in the field of sensory substitution is enabling sound
perception for deaf people by visualizing audio content. Different from
existing models that translate between speech and text or text and images, we
target immediate and low-level audio to video translation that applies to
generic environment sounds as well as human speech. Since such a substitution
is artificial, without labels for supervised learning, our core contribution is
to build a mapping from audio to video that learns from unpaired examples via
high-level constraints. For speech, we additionally disentangle content
(phones) from style (gender and dialect) by mapping them to a common
disentangled latent space. Qualitative and quantitative results, including a
user study, demonstrate that our unpaired translation approach maintains
important audio features in the generated video and that videos of faces and
numbers are well suited for visualizing high-dimensional audio features that
can be parsed by humans to match and distinguish between sounds, words, and
speakers.
