There is More than Meets the Eye: Self-Supervised Multi-Object Detection
  and Tracking with Sound by Distilling Multimodal Knowledge
  Attributes of sound inherent to objects can provide valuable cues to learn
rich representations for object detection and tracking. Furthermore, the
co-occurrence of audiovisual events in videos can be exploited to localize
objects over the image field by solely monitoring the sound in the environment.
Thus far, this has only been feasible in scenarios where the camera is static
and for single object detection. Moreover, the robustness of these methods has
been limited as they primarily rely on RGB images which are highly susceptible
to illumination and weather changes. In this work, we present the novel
self-supervised MM-DistillNet framework consisting of multiple teachers that
leverage diverse modalities including RGB, depth and thermal images, to
simultaneously exploit complementary cues and distill knowledge into a single
audio student network. We propose the new MTA loss function that facilitates
the distillation of information from multimodal teachers in a self-supervised
manner. Additionally, we propose a novel self-supervised pretext task for the
audio student that enables us to not rely on labor-intensive manual
annotations. We introduce a large-scale multimodal dataset with over 113,000
time-synchronized frames of RGB, depth, thermal, and audio modalities.
Extensive experiments demonstrate that our approach outperforms
state-of-the-art methods while being able to detect multiple objects using only
sound during inference and even while moving.
