Unsupervised Mismatch Localization in Cross-Modal Sequential Data
Content mismatch usually occurs when data from one modality is translated to
another, e.g. language learners producing mispronunciations (errors in speech)
when reading a sentence (target text) aloud. However, most existing alignment
algorithms assume the content involved in the two modalities is perfectly
matched and thus leading to difficulty in locating such mismatch between speech
and text. In this work, we develop an unsupervised learning algorithm that can
infer the relationship between content-mismatched cross-modal sequential data,
especially for speech-text sequences. More specifically, we propose a
hierarchical Bayesian deep learning model, named mismatch localization
variational autoencoder (ML-VAE), that decomposes the generative process of the
speech into hierarchically structured latent variables, indicating the
relationship between the two modalities. Training such a model is very
challenging due to the discrete latent variables with complex dependencies
involved. We propose a novel and effective training procedure which estimates
the hard assignments of the discrete latent variables over a specifically
designed lattice and updates the parameters of neural networks alternatively.
Our experimental results show that ML-VAE successfully locates the mismatch
between text and speech, without the need for human annotations for model
training.