Unsupervised Speech Enhancement using Dynamical Variational
  Auto-Encoders
  Dynamical variational autoencoders (DVAEs) are a class of deep generative
models with latent variables, dedicated to model time series of
high-dimensional data. DVAEs can be considered as extensions of the variational
autoencoder (VAE) that include temporal dependencies between successive
observed and/or latent vectors. Previous work has shown the interest of using
DVAEs over the VAE for speech spectrograms modeling. Independently, the VAE has
been successfully applied to speech enhancement in noise, in an unsupervised
noise-agnostic set-up that requires neither noise samples nor noisy speech
samples at training time, but only requires clean speech signals. In this
paper, we extend these works to DVAE-based single-channel unsupervised speech
enhancement, hence exploiting both speech signals unsupervised representation
learning and dynamics modeling. We propose an unsupervised speech enhancement
algorithm that combines a DVAE speech prior pre-trained on clean speech signals
with a noise model based on nonnegative matrix factorization, and we derive a
variational expectation-maximization (VEM) algorithm to perform speech
enhancement. The algorithm is presented with the most general DVAE formulation
and is then applied with three specific DVAE models to illustrate the
versatility of the framework. Experimental results show that the proposed
DVAE-based approach outperforms its VAE-based counterpart, as well as several
supervised and unsupervised noise-dependent baselines, especially when the
noise type is unseen during training.
