Transcription Is All You Need: Learning to Separate Musical Mixtures
  with Score as Supervision
  Most music source separation systems require large collections of isolated
sources for training, which can be difficult to obtain. In this work, we use
musical scores, which are comparatively easy to obtain, as a weak label for
training a source separation system. In contrast with previous score-informed
separation approaches, our system does not require isolated sources, and score
is used only as a training target, not required for inference. Our model
consists of a separator that outputs a time-frequency mask for each instrument,
and a transcriptor that acts as a critic, providing both temporal and frequency
supervision to guide the learning of the separator. A harmonic mask constraint
is introduced as another way of leveraging score information during training,
and we propose two novel adversarial losses for additional fine-tuning of both
the transcriptor and the separator. Results demonstrate that using score
information outperforms temporal weak-labels, and adversarial structures lead
to further improvements in both separation and transcription performance.
