Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded
  Language from Percepts and Raw Speech
  Learning to understand grounded language, which connects natural language to
percepts, is a critical research area. Prior work in grounded language
acquisition has focused primarily on textual inputs. In this work we
demonstrate the feasibility of performing grounded language acquisition on
paired visual percepts and raw speech inputs. This will allow interactions in
which language about novel tasks and environments is learned from end users,
reducing dependence on textual inputs and potentially mitigating the effects of
demographic bias found in widely available speech recognition systems. We
leverage recent work in self-supervised speech representation models and show
that learned representations of speech can make language grounding systems more
inclusive towards specific groups while maintaining or even increasing general
performance.
