Active Audio-Visual Separation of Dynamic Sound Sources
  We explore active audio-visual separation for dynamic sound sources, where an
embodied agent moves intelligently in a 3D environment to continuously isolate
the time-varying audio stream being emitted by an object of interest. The agent
hears a mixed stream of multiple time-varying audio sources (e.g., multiple
people conversing and a band playing music at a noisy party). Given a limited
time budget, it needs to extract the target sound using egocentric audio-visual
observations. We propose a reinforcement learning agent equipped with a novel
transformer memory that learns motion policies to control its camera and
microphone to recover the dynamic target audio, improving its own estimates for
past timesteps via self-attention. Using highly realistic acoustic SoundSpaces
simulations in real-world scanned Matterport3D environments, we show that our
model is able to learn efficient behavior to carry out continuous separation of
a time-varying audio target. Project:
https://vision.cs.utexas.edu/projects/active-av-dynamic-separation/.
