Multi-view Temporal Alignment for Non-parallel Articulatory-to-Acoustic
  Speech Synthesis
  Articulatory-to-acoustic (A2A) synthesis refers to the generation of audible
speech from captured movement of the speech articulators. This technique has
numerous applications, such as restoring oral communication to people who
cannot longer speak due to illness or injury. Most successful techniques so far
adopt a supervised learning framework, in which time-synchronous
articulatory-and-speech recordings are used to train a supervised machine
learning algorithm that can be used later to map articulator movements to
speech. This, however, prevents the application of A2A techniques in cases
where parallel data is unavailable, e.g., a person has already lost her/his
voice and only articulatory data can be captured. In this work, we propose a
solution to this problem based on the theory of multi-view learning. The
proposed algorithm attempts to find an optimal temporal alignment between pairs
of non-aligned articulatory-and-acoustic sequences with the same phonetic
content by projecting them into a common latent space where both views are
maximally correlated and then applying dynamic time warping. Several variants
of this idea are discussed and explored. We show that the quality of speech
generated in the non-aligned scenario is comparable to that obtained in the
parallel scenario.
