Ecological Semantics: Programming Environments for Situated Language
  Understanding
  Large-scale natural language understanding (NLU) systems have made impressive
progress: they can be applied flexibly across a variety of tasks, and employ
minimal structural assumptions. However, extensive empirical research has shown
this to be a double-edged sword, coming at the cost of shallow understanding:
inferior generalization, grounding and explainability. Grounded language
learning approaches offer the promise of deeper understanding by situating
learning in richer, more structured training environments, but are limited in
scale to relatively narrow, predefined domains. How might we enjoy the best of
both worlds: grounded, general NLU? Following extensive contemporary cognitive
science, we propose treating environments as "first-class citizens" in semantic
representations, worthy of research and development in their own right.
Importantly, models should also be partners in the creation and configuration
of environments, rather than just actors within them, as in existing
approaches. To do so, we argue that models must begin to understand and program
in the language of affordances (which define possible actions in a given
situation) both for online, situated discourse comprehension, as well as
large-scale, offline common-sense knowledge mining. To this end we propose an
environment-oriented ecological semantics, outlining theoretical and practical
approaches towards implementation. We further provide actual demonstrations
building upon interactive fiction programming languages.
