Real-time Timbre Transfer and Sound Synthesis using DDSP
  Neural audio synthesis is an actively researched topic, having yielded a wide
range of techniques that leverages machine learning architectures. Google
Magenta elaborated a novel approach called Differential Digital Signal
Processing (DDSP) that incorporates deep neural networks with preconditioned
digital signal processing techniques, reaching state-of-the-art results
especially in timbre transfer applications. However, most of these techniques,
including the DDSP, are generally not applicable in real-time constraints,
making them ineligible in a musical workflow. In this paper, we present a
real-time implementation of the DDSP library embedded in a virtual synthesizer
as a plug-in that can be used in a Digital Audio Workstation. We focused on
timbre transfer from learned representations of real instruments to arbitrary
sound inputs as well as controlling these models by MIDI. Furthermore, we
developed a GUI for intuitive high-level controls which can be used for
post-processing and manipulating the parameters estimated by the neural
network. We have conducted a user experience test with seven participants
online. The results indicated that our users found the interface appealing,
easy to understand, and worth exploring further. At the same time, we have
identified issues in the timbre transfer quality, in some components we did not
implement, and in installation and distribution of our plugin. The next
iteration of our design will address these issues. Our real-time MATLAB and
JUCE implementations are available at https://github.com/SMC704/juce-ddsp and
https://github.com/SMC704/matlab-ddsp , respectively.
