Effective Use of Variational Embedding Capacity in Expressive End-to-End
  Speech Synthesis
  Recent work has explored sequence-to-sequence latent variable models for
expressive speech synthesis (supporting control and transfer of prosody and
style), but has not presented a coherent framework for understanding the
trade-offs between the competing methods. In this paper, we propose embedding
capacity (the amount of information the embedding contains about the data) as a
unified method of analyzing the behavior of latent variable models of speech,
comparing existing heuristic (non-variational) methods to variational methods
that are able to explicitly constrain capacity using an upper bound on
representational mutual information. In our proposed model (Capacitron), we
show that by adding conditional dependencies to the variational posterior such
that it matches the form of the true posterior, the same model can be used for
high-precision prosody transfer, text-agnostic style transfer, and generation
of natural-sounding prior samples. For multi-speaker models, Capacitron is able
to preserve target speaker identity during inter-speaker prosody transfer and
when drawing samples from the latent prior. Lastly, we introduce a method for
decomposing embedding capacity hierarchically across two sets of latents,
allowing a portion of the latent variability to be specified and the remaining
variability sampled from a learned prior. Audio examples are available on the
web.
