Synchronising audio and ultrasound by learning cross-modal embeddings
  Audiovisual synchronisation is the task of determining the time offset
between speech audio and a video recording of the articulators. In child speech
therapy, audio and ultrasound videos of the tongue are captured using
instruments which rely on hardware to synchronise the two modalities at
recording time. Hardware synchronisation can fail in practice, and no mechanism
exists to synchronise the signals post hoc. To address this problem, we employ
a two-stream neural network which exploits the correlation between the two
modalities to find the offset. We train our model on recordings from 69
speakers, and show that it correctly synchronises 82.9% of test utterances from
unseen therapy sessions and unseen speakers, thus considerably reducing the
number of utterances to be manually synchronised. An analysis of model
performance on the test utterances shows that directed phone articulations are
more difficult to automatically synchronise compared to utterances containing
natural variation in speech such as words, sentences, or conversations.
