Enhancing Zero-Shot Many to Many Voice Conversion with Self-Attention
  VAE
  Variational auto-encoder(VAE) is an effective neural network architecture to
disentangle a speech utterance into speaker identity and linguistic content
latent embeddings, then generate an utterance for a target speaker from that of
a source speaker. This is possible by concatenating the identity embedding of
the target speaker and the content embedding of the source speaker uttering a
desired sentence. In this work, we found a suitable location of VAE's decoder
to add a self-attention layer for incorporating non-local information in
generating a converted utterance and hiding the source speaker's identity. In
experiments of zero-shot many-to-many voice conversion task on VCTK data set,
the self-attention layer enhances speaker classification accuracy on unseen
speakers by 27\% while increasing the decoder parameter size by 12\%. The voice
quality of converted utterance degrades by merely 3\% measured by the MOSNet
scores. To reduce over-fitting and generalization error, we further applied a
relaxed group-wise splitting method in network training and achieved a gain of
speaker classification accuracy on unseen speakers by 46\% while maintaining
the conversion voice quality in terms of MOSNet scores. Our encouraging
findings point to future research on integrating more variety of attention
structures in VAE framework for advancing zero-shot many-to-many voice
conversions.
