Data Fusion for Audiovisual Speaker Localization: Extending Dynamic
  Stream Weights to the Spatial Domain
  Estimating the positions of multiple speakers can be helpful for tasks like
automatic speech recognition or speaker diarization. Both applications benefit
from a known speaker position when, for instance, applying beamforming or
assigning unique speaker identities. Recently, several approaches utilizing
acoustic signals augmented with visual data have been proposed for this task.
However, both the acoustic and the visual modality may be corrupted in specific
spatial regions, for instance due to poor lighting conditions or to the
presence of background noise. This paper proposes a novel audiovisual data
fusion framework for speaker localization by assigning individual dynamic
stream weights to specific regions in the localization space. This fusion is
achieved via a neural network, which combines the predictions of individual
audio and video trackers based on their time- and location-dependent
reliability. A performance evaluation using audiovisual recordings yields
promising results, with the proposed fusion approach outperforming all baseline
models.
