Learning to Transcribe by Ear
  Rethinking how to model polyphonic transcription formally, we frame it as a
reinforcement learning task. Such a task formulation encompasses the notion of
a musical agent and an environment containing an instrument as well as the
sound source to be transcribed. Within this conceptual framework, the
transcription process can be described as the agent interacting with the
instrument in the environment, and obtaining reward by playing along with what
it hears. Choosing from a discrete set of actions - the notes to play on its
instrument - the amount of reward the agent experiences depends on which notes
it plays and when. This process resembles how a human musician might approach
the task of transcription, and the satisfaction she achieves by closely
mimicking the sound source to transcribe on her instrument. Following a
discussion of the theoretical framework and the benefits of modelling the
problem in this way, we focus our attention on several practical considerations
and address the difficulties in training an agent to acceptable performance on
a set of tasks with increasing difficulty. We demonstrate promising results in
partially constrained environments.
