Can We Trust Deep Speech Prior?
  Recently, speech enhancement (SE) based on deep speech prior has attracted
much attention, such as the variational auto-encoder with non-negative matrix
factorization (VAE-NMF) architecture. Compared to conventional approaches that
represent clean speech by shallow models such as Gaussians with a low-rank
covariance, the new approach employs deep generative models to represent the
clean speech, which often provides a better prior. Despite the clear advantage
in theory, we argue that deep priors must be used with much caution, since the
likelihood produced by a deep generative model does not always coincide with
the speech quality. We designed a comprehensive study on this issue and
demonstrated that based on deep speech priors, a reasonable SE performance can
be achieved, but the results might be suboptimal. A careful analysis showed
that this problem is deeply rooted in the disharmony between the flexibility of
deep generative models and the nature of the maximum-likelihood (ML) training.
