End-to-End Speech Separation with Unfolded Iterative Phase
  Reconstruction
  This paper proposes an end-to-end approach for single-channel
speaker-independent multi-speaker speech separation, where time-frequency (T-F)
masking, the short-time Fourier transform (STFT), and its inverse are
represented as layers within a deep network. Previous approaches, rather than
computing a loss on the reconstructed signal, used a surrogate loss based on
the target STFT magnitudes. This ignores reconstruction error introduced by
phase inconsistency. In our approach, the loss function is directly defined on
the reconstructed signals, which are optimized for best separation. In
addition, we train through unfolded iterations of a phase reconstruction
algorithm, represented as a series of STFT and inverse STFT layers. While mask
values are typically limited to lie between zero and one for approaches using
the mixture phase for reconstruction, this limitation is less relevant if the
estimated magnitudes are to be used together with phase reconstruction. We thus
propose several novel activation functions for the output layer of the T-F
masking, to allow mask values beyond one. On the publicly-available wsj0-2mix
dataset, our approach achieves state-of-the-art 12.6 dB scale-invariant
signal-to-distortion ratio (SI-SDR) and 13.1 dB SDR, revealing new
possibilities for deep learning based phase reconstruction and representing a
fundamental progress towards solving the notoriously-hard cocktail party
problem.
