Progressive Learning for Stabilizing Label Selection in Speech
  Separation with Mapping-based Method
  Speech separation has been studied in time domain because of lower latency
and higher performance compared to time-frequency domain. The masking-based
method has been mostly used in time domain, and the other common method
(mapping-based) has been inadequately studied. We investigate the use of the
mapping-based method in the time domain and show that it can perform better on
a large training set than the masking-based method. We also investigate the
frequent label-switching problem in permutation invariant training (PIT), which
results in suboptimal training because the labels selected by PIT differ across
training epochs. Our experiment results showed that PIT works well in a shallow
separation model, and the label switching occurs for a deeper model. We
inferred that layer decoupling may be the reason for the frequent label
switching. Therefore, we propose a training strategy based on progressive
learning. This approach significantly reduced inconsistent label assignment
without added computational complexity or training corpus. By combining this
training strategy with the mapping-based method, we significantly improved the
separation performance compared to the baseline.
