Sequential Multi-Frame Neural Beamforming for Speech Separation and
  Enhancement
  This work introduces sequential neural beamforming, which alternates between
neural network based spectral separation and beamforming based spatial
separation. Our neural networks for separation use an advanced convolutional
architecture trained with a novel stabilized signal-to-noise ratio loss
function. For beamforming, we explore multiple ways of computing time-varying
covariance matrices, including factorizing the spatial covariance into a
time-varying amplitude component and a time-invariant spatial component, as
well as using block-based techniques. In addition, we introduce a multi-frame
beamforming method which improves the results significantly by adding
contextual frames to the beamforming formulations. We extensively evaluate and
analyze the effects of window size, block size, and multi-frame context size
for these methods. Our best method utilizes a sequence of three neural
separation and multi-frame time-invariant spatial beamforming stages, and
demonstrates an average improvement of 2.75 dB in scale-invariant
signal-to-noise ratio and 14.2% absolute reduction in a comparative speech
recognition metric across four challenging reverberant speech enhancement and
separation tasks. We also use our three-speaker separation model to separate
real recordings in the LibriCSS evaluation set into non-overlapping tracks, and
achieve a better word error rate as compared to a baseline mask based
beamformer.
