{'fraction_non_alphanumeric': 0.021645569620253165, 'fraction_numerical': 0.005063291139240506, 'mean_word_length': 5.143856920684293, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Paradigm for Situated and Goal-Driven Language Learning

A distinguishing property of human intelligence is the ability to flexibly use language in order to communicate complex ideas with other humans in a variety of contexts. Research in natural language dialogue should focus on designing communicative agents which can integrate themselves into these contexts and productively collaborate with humans. In this abstract, we propose a general situated language learning paradigm which is designed to bring about robust language agents able to cooperate productively with humans.


The environment
We propose an end-to-end learning environment with multiple language-enabled agents, each with the capacity to define their own internal goals and plans to reach those goals. Each agent may also have different capacities to observe or act in this environment. Their goals are grounded non-linguistic objectives: for example, to reach a desired location, manipulate objects in the environment, or transmit a piece of information. 2 (We address the issue of non-linguistic grounding at length in the next section.) Some fixed-language agents in the environment speak an existing conventional language (e.g. English), and other learning agents are tasked with jointly learning this language while also solving other goals in the environment. The agents are assigned difficult (possibly distinct) tasks which 1 Our motivation here is similar to that of Dagan et al. (2006), who recognized the negative effects of a community fragmented across different isolated application-specific tasks, and suggested the unified task of recognizing textual entailment (RTE) as a solution.
2 A related line of work in evolutionary linguistics constructs a similar language learning scenario entirely without fixed-language agents (Smith et al., 2003;Steels, 2012;Kirby et al., 2014). All of the agents in these environments construct a novel language simultaneously to accomplish some shared task. This is an interesting separate line of research, but ultimately a separate task from the understanding and acquisition problems discussed in this abstract. require them to cooperate by communicating via their language channel. 3 In the simplest instance of the paradigm above, a single fixed-language parent agent interacts with a learning child agent. The parent cooperates with the child, but only when prompted through language. As a result, the child acquires the language of the parent in order to accomplish its task. This specific scenario is designed to train single communicative agents which can accomplish tasks in their environments with the help of human or simulated counterparts.
It is possible to include fixed-language agents in these environments without requiring human involvement. We have developed several instantiations of this paradigm in which the fixed-language agents speak some simplified hard-coded English or an artificial language (e.g. a programming language or query language). Near the limit of artificial complexity, agents might take advantage of the Internet to access world knowledge and synthesize responses. As our algorithms for learning in these environments improve, however, we expect that involving human agents in the environment would be an extremely effective way to train language-enabled artificial agents.

Environment grounding
Our language-learning agents crucially need to be grounded in a world which is not only linguistic. This grounding may be physical -for example, agents which are embodied in the real world -or virtual. This grounding is what allows us to evaluate the agents in some way that does not prioritize language as the only objective. Grounding imposes additional responsibilities on the agent, such as competently sensing and acting in the world, but this should be seen as an opportunity rather than a problem. As we discuss in the following paragraphs, a grounded agent may use its experience in its environment to build better models of language, and likewise use its language to better reason about its environment.
This grounded environment is designed to bring about agents with comprehensive predictive models of the world, which combine linguistic knowledge with more general intelligent behavior. By design, we do not separate the activity of language model construction from many other intelligent predictive activities -whether physical (predicting physical behavior of objects), psychological (modeling the beliefs and intentions of other agents), or social (understanding group membership and group-level action) (Lake et al., 2016;Clark, 2013). While early instantiations of this environment will limit the complexity necessary for the agents to model, we expect all of these different factors to eventually be relevant in a comprehensive learning environment. Mikolov et al. (2015) make a similar argument for grounding, but arrive at an environment in which perception and action is mediated only through a linguistic channel. While this textual interface certainly has the potential to simplify the problem, we believe there are several issues with this setup.
Textual interfaces impose a lossy representation of an agent's actual environment (Brooks, 1991), and necessarily bias the focus of an agent's observations. Consider the following text-only interaction in the style of Mikolov et al. (2015) between a learning agent A and omniscient agent B, situated in some simulated physical world: The agent could have avoided this disaster if it had queried to find out that the table was missing a leg before placing weight on it. But we cannot expect an agent to exhaustively query its environment via text in general. (Should it also ask whether the table is made of solid material, or whether the table is on fire?) By contrast, a lossless visual observation does not remove possibly relevant information about the properties of tables, and it is up to the agent to learn what to attend to in such an environment and how to interpret these visual percepts.
For environments involving pre-programmed fixed-language agents, generating textual environment descriptions also places a significant implementational burden. It is relatively easy to generate physical motion and visual renderings of three-legged tables or toppling events, but much more difficult to describe these concepts in natural language.
Of course, even if we claim that text data is not enough, we must settle for some level of abstraction. A similar argument to the one above could be made for including sound-wave speech data in the agent's observations rather than only providing text and visual data. While this argument is likely also correct, we have to settle for a level of representation which is computationally tractable. Given that we want to make near-term progress on language comprehension and production, it seems reasonable to work with messages in text representation for the time being. It is likewise tractable to work in a virtual environment with simulated physics and visual inputs.

Conclusion
In this abstract, we outlined a paradigm for grounded and goal-driven language learning in artificial agents. The paradigm is centered around a utilitarian definition of language understanding, which equates language understanding with the ability to cooperate with other language users in real-world environments. This position demotes language from its position as a separate task to be solved to one of several communicative tools agents might use to accomplish their real-world goals.
While this paradigm does already capture a small amount of recent work in dialogue, on the whole it has not received the focus it deserves in the research communities of natural language processing and machine learning. We hope this paper brings focus to the task of situated language learning as a way forward for research in natural language dialogue.