CAESynth: Real-Time Timbre Interpolation and Pitch Control with
  Conditional Autoencoders
  In this paper, we present a novel audio synthesizer, CAESynth, based on a
conditional autoencoder. CAESynth synthesizes timbre in real-time by
interpolating the reference sounds in their shared latent feature space, while
controlling a pitch independently. We show that training a conditional
autoencoder based on accuracy in timbre classification together with
adversarial regularization of pitch content allows timbre distribution in
latent space to be more effective and stable for timbre interpolation and pitch
conditioning. The proposed method is applicable not only to creation of musical
cues but also to exploration of audio affordance in mixed reality based on
novel timbre mixtures with environmental sounds. We demonstrate by experiments
that CAESynth achieves smooth and high-fidelity audio synthesis in real-time
through timbre interpolation and independent yet accurate pitch control for
musical cues as well as for audio affordance with environmental sound. A Python
implementation along with some generated samples are shared online.
