Improving Ultrasound Tongue Image Reconstruction from Lip Images Using
  Self-supervised Learning and Attention Mechanism
  Speech production is a dynamic procedure, which involved multi human organs
including the tongue, jaw and lips. Modeling the dynamics of the vocal tract
deformation is a fundamental problem to understand the speech, which is the
most common way for human daily communication. Researchers employ several
sensory streams to describe the process simultaneously, which are
incontrovertibly statistically related to other streams. In this paper, we
address the following question: given an observable image sequences of lips,
can we picture the corresponding tongue motion. We formulated this problem as
the self-supervised learning problem, and employ the two-stream convolutional
network and long-short memory network for the learning task, with the attention
mechanism. We evaluate the performance of the proposed method by leveraging the
unlabeled lip videos to predict an upcoming ultrasound tongue image sequence.
The results show that our model is able to generate images that close to the
real ultrasound tongue images, and results in the matching between two imaging
modalities.
