Generalization Challenges for Neural Architectures in Audio Source
  Separation
  Recent work has shown that recurrent neural networks can be trained to
separate individual speakers in a sound mixture with high fidelity. Here we
explore convolutional neural network models as an alternative and show that
they achieve state-of-the-art results with an order of magnitude fewer
parameters. We also characterize and compare the robustness and ability of
these different approaches to generalize under three different test conditions:
longer time sequences, the addition of intermittent noise, and different
datasets not seen during training. For the last condition, we create a new
dataset, RealTalkLibri, to test source separation in real-world environments.
We show that the acoustics of the environment have significant impact on the
structure of the waveform and the overall performance of neural network models,
with the convolutional model showing superior ability to generalize to new
environments. The code for our study is available at
https://github.com/ShariqM/source_separation.
