Grounding Spatio-Temporal Language with Transformers
  Language is an interface to the outside world. In order for embodied agents
to use it, language must be grounded in other, sensorimotor modalities. While
there is an extended literature studying how machines can learn grounded
language, the topic of how to learn spatio-temporal linguistic concepts is
still largely uncharted. To make progress in this direction, we here introduce
a novel spatio-temporal language grounding task where the goal is to learn the
meaning of spatio-temporal descriptions of behavioral traces of an embodied
agent. This is achieved by training a truth function that predicts if a
description matches a given history of observations. The descriptions involve
time-extended predicates in past and present tense as well as spatio-temporal
references to objects in the scene. To study the role of architectural biases
in this task, we train several models including multimodal Transformer
architectures; the latter implement different attention computations between
words and objects across space and time. We test models on two classes of
generalization: 1) generalization to randomly held-out sentences; 2)
generalization to grammar primitives. We observe that maintaining object
identity in the attention computation of our Transformers is instrumental to
achieving good performance on generalization overall, and that summarizing
object traces in a single token has little influence on performance. We then
discuss how this opens new perspectives for language-guided autonomous embodied
agents. We also release our code under open-source license as well as
pretrained models and datasets to encourage the wider community to build upon
and extend our work in the future.
