Additional Shared Decoder on Siamese Multi-view Encoders for Learning
  Acoustic Word Embeddings
  Acoustic word embeddings --- fixed-dimensional vector representations of
arbitrary-length words --- have attracted increasing interest in
query-by-example spoken term detection. Recently, on the fact that the
orthography of text labels partly reflects the phonetic similarity between the
words' pronunciation, a multi-view approach has been introduced that jointly
learns acoustic and text embeddings. It showed that it is possible to learn
discriminative embeddings by designing the objective which takes text labels as
well as word segments. In this paper, we propose a network architecture that
expands the multi-view approach by combining the Siamese multi-view encoders
with a shared decoder network to maximize the effect of the relationship
between acoustic and text embeddings in embedding space. Discriminatively
trained with multi-view triplet loss and decoding loss, our proposed approach
achieves better performance on acoustic word discrimination task with the WSJ
dataset, resulting in 11.1% relative improvement in average precision. We also
present experimental results on cross-view word discrimination and word level
speech recognition tasks.
