On Using Transformers for Speech-Separation
  Transformers have enabled major improvements in deep learning. They often
outperform recurrent and convolutional models in many tasks while taking
advantage of parallel processing. Recently, we have proposed SepFormer, which
uses self-attention and obtains state-of-the art results on WSJ0-2/3 Mix
datasets for speech separation. In this paper, we extend our previous work by
providing results on more datasets including LibriMix, and WHAM!, WHAMR! which
include noisy and noisy-reverberant conditions. Moreover we provide denoising,
and denoising+dereverberation results in the context of speech enhancement,
respectively on WHAM! and WHAMR! datasets. We also investigate incorporating
recently proposed efficient self-attention mechanisms inside the SepFormer
model, and show that by using efficient self-attention mechanisms it is
possible to reduce the memory requirements significantly while performing
better than the popular convtasnet model on WSJ0-2Mix dataset.
