Talk the Walk: Navigating New York City through Grounded Dialogue
  We introduce "Talk The Walk", the first large-scale dialogue dataset grounded
in action and perception. The task involves two agents (a "guide" and a
"tourist") that communicate via natural language in order to achieve a common
goal: having the tourist navigate to a given target location. The task and
dataset, which are described in detail, are challenging and their full solution
is an open problem that we pose to the community. We (i) focus on the task of
tourist localization and develop the novel Masked Attention for Spatial
Convolutions (MASC) mechanism that allows for grounding tourist utterances into
the guide's map, (ii) show it yields significant improvements for both emergent
and natural language communication, and (iii) using this method, we establish
non-trivial baselines on the full task.
