{'fraction_non_alphanumeric': 0.03259572895139204, 'fraction_numerical': 0.014365643429608348, 'mean_word_length': 4.911601112435439, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DECSTR: Learning Goal-Directed Abstract Behaviors using Pre-Verbal Spatial Predicates in Intrinsically Motivated Agents

Intrinsically motivated agents freely explore their environment and set their own goals. Such goals are traditionally represented as specific states, but recent works introduced the use of language to facilitate abstraction. Language can, for example, represent goals as sets of general properties that surrounding objects should verify. However, language-conditioned agents are trained simultaneously to understand language and to act, which seems to contrast with how children learn: infants demonstrate goal-oriented behaviors and abstract spatial concepts very early in their development, before language mastery. Guided by these findings from developmental psychology, we introduce a high-level state representation based on natural semantic predicates that describe spatial relations between objects and that are known to be present early in infants. In a robotic manipulation environment, our DECSTR system explores this representation space by manipulating objects, and efficiently learns to achieve any reachable configuration within it. It does so by leveraging an object-centered modular architecture, a symmetry inductive bias, and a new form of automatic curriculum learning for goal selection and policy learning. As with children, language acquisition takes place in a second phase, independently from goal-oriented sensorimotor learning. This is done via a new goal generation module, conditioned on instructions describing expected transformations in object relations. We present ablations studies for each component and highlight several advantages of targeting abstract goals over specific ones. We further show that using this intermediate representation enables efficient language grounding by evaluating agents on sequences of language instructions and their logical combinations.


Introduction
Developmental psychology investigates the interactions between learning and developmental processes that support the slow but extraordinary transition from the mental life of infants to the sophisticated intelligence of human adults Piaget [1977], Smith and Gasser [2005]. Inspired by this line of thought, the central endeavour of developmental robotics consists in shaping a set of machine learning processes able to generate a similar growth of capabilities in robots Weng et al. [2001], Lungarella et al. [2003]. This approach is particularly interested in the Symbol Grounding Problem Harnad [1990], the challenge of explaining how a machine can extract the meaning of symbols by grounding them into non-symbolic sensorimotor interactions Sigaud and Droniou [2016].
The design of intrinsically motivated agents marked a major step in developmental robotics. These embodied agents interact with their environment at the sensorimotor level and are provided with the ability to represent and set their own goals, rewarding themselves over completion [Forestier et al., 2017]. They often come with the capacity to automatically organize their learning curriculum Portelas et al. [2020], deciding which goals to target and learn about as a function of their current abilities Florensa et al. [2017], Fournier et al. [2019], Colas et al. [2019a], Racaniere et al. [2019]. In these works, just as in most goal-conditioned reinforcement learning (RL) algorithms Schaul et al. [2015], Andrychowicz et al. [2017], the goal space is generally defined as a subset of the state space of the agents, including when it is learned Nair et al. [2018], Florensa et al. [2018], Pong et al. [2019], Nair et al. [2019]. This approach falls short of addressing the Symbol Grounding Problem: it does not provide a level of abstraction that facilitates the communication with other artificial or human agents about these goals, which may be necessary for symbolic intelligence Bruner [2009], Taniguchi et al. [2018].
Language, because of its capacity of abstraction, is a natural candidate in the quest for the higher-level goal representations required to solve the Symbol Grounding Problem Cangelosi et al. [2010]. This connection between sensorimotor behavior and language has recently emerged in the RL community under the form of language-conditioned agents Chan et al. [2019], Bahdanau et al. [2018], Cideron et al. [2019], Jiang et al. [2019], Luketina et al. [2019], Colas et al. [2020]. However, all these approaches use language as a necessary input to sensorimotor behavior and, for this reason, cannot account for the goal-directed behaviors observed in pre-verbal infants Wood et al. [1976]. Indeed, developmental psychology has shown that, in addition to object-centered representations Spelke and Kinzler [2007], pre-verbal infants were equipped with a small set of conceptual relational spatial primitives Mandler [2012]. These forms of predicates over object relations enable pre-verbal abstract thoughts and, later on, support early language learning Mandler [2012].
Based on these findings, and to circumvent the limitations of the direct mapping between language and behavior that is ubiquitous in instruction following agents, we propose DECSTR 2 . DECSTR is a learning architecture that trains intrinsically motivated agents to explore a semantic representation space characterizing spatial relations between physical objects. Our agents have access to both object-centered features and a few spatial relation predicates. In a first phase, DECSTR learns to discover and master all reachable configurations in their semantic representation space. In a second phase, DECSTR agents learn to ground a simplified descriptive language into their internal semantic representations and their novel set of skills.
To summarize, this paper introduces the following contributions: 1) the concept of leveraging pre-verbal semantic representations for the acquisition of abstract behavioral repertoires; 2) a new approach to language grounding that can be decoupled from sensorimotor learning. These conceptual contributions are supported by technical contributions: 1) new object-centered inductive biases; 2) a novel automatic curriculum approach and 3) a language-conditioned goal generation module. This study is conducted in the Fetch Manipulate environment, where intrinsically motivated agents are embodied in a simulated robotic arm facing blocks, and must learn to manipulate them to discover and master various configurations from the semantic representation space.

Related work
From language-conditioned policies to predicate-based policies. Language was recently introduced as a way to help or instruct goal-conditioned learning agents [Luketina et al., 2019]. Most of these works consider instruction-following agents [Hermann et al., 2017, Chan et al., 2019, Bahdanau et al., 2018, Cideron et al., 2019, Jiang et al., 2019, Fu et al., 2019, whereas the Imagine approach introduced intrinsically motivated agents able to set their own goals and to imagine new ones [Colas et al., 2020]. Other approaches used descriptive language to build representations Waytowich et al. [2019] or to characterize the dynamics of the environment Zhong et al. [2019]. Our approach is different: because the policy is not conditioned on language, we can decouple language grounding from the acquisition of sensorimotor skills. Close to us, Kulick et al. [2013] address the acquisition of symbolic representations based on geometric relations between objects in a robotic interaction scenario. Starting with pre-trained robots, they focus on learning a mapping between pre-acquired behaviors and pre-existing symbols. This aims to simplify the user-robot interactions, but does not leverage natural language. Moreover, these symbols are used in a relational RL framework Džeroski et al. [2001], not to form goals for goal-conditioned policies.
Achieving complex goals by manipulating blocks. Stacking blocks is one of the earliest benchmarks in artificial intelligence (e.g. Sussman [1973], Tate et al. [1975]) and has led to many simulation and robotics studies, e.g. [Deisenroth et al., 2011, Xu et al., 2018, Colas et al., 2019a. Like us, several recent papers have addressed complex block manipulation in the Fetch OpenAI environment [Plappert et al., 2018]. Lanier et al. [2019] achieve stacks of 4 blocks, whereas Li et al.
[2019] achieve a variety of goals with up to 6 blocks. Both works rely on heavy hand-defined curriculum strategies and represent goals as a set of specific 3D target positions for each block. No matter the abstract configuration a goal refers to, agents only need to learn to move blocks to their target positions. In contrast, the use of semantic goal representations is more flexible: the agent can decide on the locations of the blocks, as long as the spatial relations described by the semantic goal are verified.
Automated curriculum learning for the acquisition of behavioral repertoires. Li et al. [2019] use a hand-defined curriculum where additional blocks are added to the experiment once the performance of the agent passes a specified threshold. In contrast, our agents learn their own curriculum. The main principle behind curriculum learning is to organize tasks from simple to hard to facilitate learning and improve transfer Elman [1993]. Automatic curriculum learning is currently the focus of a variety of works Matiisen et al. [2019], Portelas et al. [2020. Among these methods, several choose to bias goal sampling towards goals associated to high-learning progress [Colas et al., 2019a, Fournier et al., 2019. The corresponding agents track estimations of learning progress over sets of goals to improve robustness. So far, these sets were hand-defined. Removing that constraint, we propose to learn them online by grouping goals according to their time of discovery.
Relational inductive biases Previous work has advocated for the use of inductive biases in learning architectures Battaglia et al. [2018]. Relational inductive biases in particular, can help learning agents to interact with objects characterized by their relations. Colas et al. [2020] first presented an objectcentered modular architecture in RL to improve transfer of skills between objects that was then extended to consider object pairs in reward prediction Karch et al. [2020]. Here, we leverage the symmetry properties of some spatial relations as additional inductive biases into our learning architecture.

Methods
We consider agents equipped with innate semantic representations of their states, evolving in an unknown environment containing physical objects. These semantic representations can be formalized as a list of spatial relations between objects in the scene. In a first sensorimotor learning phase, the agent must explore its innate semantic space, discover reachable configurations and learn to robustly reproduce them. In a second language grounding phase, it must learn to relate sentences from a social partner to its newly acquired set of skills.
To tackle this problem, we propose the DECSTR learning architecture. In the sensorimotor learning phase, the agent leverages an automatic curriculum mechanism to sample from the set of known goals, boostrapping with random configurations. It alternates between goal-directed interactions with objects and actual learning updates. It updates both the list of known goals and the sampling probabilities of its goal sampler, as well as its policy using the Soft Actor-Critic algorithm Haarnoja et al. [2018] and Hindsight Experience Replay Andrychowicz et al. [2017]. In the language grounding phase, DECSTR receives language descriptions of its goal-directed behavior and uses them to train a language-conditioned goal generation module that links language to skills, effectively grounding language in sensorimotor behavior.
This section introduces Fetch Manipulate, an environment designed to study the acquisition of language-grounded abstract behaviors (Section 3.1). It also presents the novel contributions of the DECSTR algorithm: new forms of object-centered inductive biases (Section 3.2), a novel automatic curriculum strategy (Section 3.3) and a new type of language module that enables language grounding (Section 3.4).

Abstract Semantic Representations in the Fetch Manipulation Environment
Semantic configurations. We assume that agents have access to innate representations called semantic configurations. These configurations are based on a list of predicates describing spatial relations between pairs of objects in the scene. In this paper, we consider two of the spatial predicates infants demonstrate early in their development Mandler [2012]: the close and the above binary predicates. These two predicates are applied to all permutations of object pairs, i.e. 6 permutations for the 3 objects we consider. Because the close predicate is order invariant, we only need to evaluate it on 3 object combinations. The above predicate being order dependent, we need all 6 permutations. The resulting binary vector of size 9 forms the semantic configuration. It represents the spatial relations between objects in the scene. In the resulting semantic configuration space {0, 1} 9 , the agent can reach 35 physically valid configurations, including stacks of 2 or 3 blocks and pyramids. Note that, in principle, the DECSTR learning architecture could use any other combination of binary predicates and could be extended to use n-ary predicates. Supplementary Section 6 provides formal definitions and properties of predicates and semantic configurations. The Fetch Manipulate environment. To evaluate DECSTR, we introduce the Fetch Manipulate environment: a robotic manipulation domain based on MUJOCO Todorov et al. [2012] and adapted from the Fetch tasks Plappert et al. [2018]. A 4 DoF robotic arm faces 3 colored blocks. In contrast with traditional approaches, goals are not defined as particular targets for each block but as semantic configurations. The input of the agent is augmented with the current semantic configuration computed from the current state. The binary reward function directly derives from the semantic mapping: the agent rewards itself when its current configuration c p matches the goal configuration c p = g. Supplementary Section 7 provides illustrated examples of valid configurations.

Object-Centered Inductive Biases
DECSTR leverages two types of object-centered inductive biases in the design of its policy and critic.
Inductive bias for efficient skill transfer between object pairs. The first inductive bias lies in the design of the networks themselves. A shared network independently encodes affordances between the agent (body features) and each pair of objects (object features), as well as the current and goal configurations. This is inspired from architectures presented in Colas et al. [2020], Karch et al. [2020], based on Deep Sets Zaheer et al. [2017]. This shared network ensures efficient transfer of skills between object pairs.
Inductive bias for efficient skill transfer between symmetric behaviors. The second inductive bias leverages the symmetry of the behavior required to achieve above(o i , o j ) and above(o j , o i ). Encoding separately both predicates in the goal configurations would force the shared network to learn about these two predicates independently. To ensure automatic transfer between the two, we present half of the affordances (e.g. those based on pairs (o i , o j ) where i < j) with goals containing one side of the symmetry (all above(o i , o j ) for i < j) and the other half of the affordances with the goals containing the other side of the symmetry (all above(o j , o i ) for i < j). As a result, the above(o i , o j ) predicates fall into the same slot of the shared network inputs as their symmetric counterparts above(o j , o i ), only with different permutations of object pairs. The goals are now of size 6: 3 close and 3 above corresponding to one side of the above symmetry. Skill transfer between symmetric predicates are automatically ensured. Supplementary Section 8.2 further describes these inductive biases and represents our modular architectures.

Curriculum Learning
DECSTR agents use an automatic curriculum strategy Portelas et al. [2020] inspired from the CURIOUS algorithm Colas et al. [2019a]. They track aggregated estimations of their competence (C), learning progress (LP). They bias their selection of goals to target during data collection and goals to learn about during policy updates towards goals associated with high absolute LP and low C.
Automatic bucket generation. To facilitate robust estimations, LP is usually estimated on sets of goals with similar characteristics (e.g. similar difficulty, similar dynamics) Forestier et al. [2017], Colas et al. [2019a]. While previous works leveraged expert-defined goal buckets, we propose an automatic strategy that groups goals together based on their time of discovery. The intuition is that the time of discovery is a good proxy for goal difficulty: easy goals are discovered early, more difficult goals later. Agents start with no known configurations (empty buckets). When they end an episode in a new configuration, the N b = 5 buckets are updated. Buckets are filled equally and the first buckets contain the configurations discovered earlier. For this reason, goals change buckets as new goals are discovered.
Tracking competence, learning progress and sampling probabilities. Regularly, agents perform self-evaluations: they evaluate themselves on goal configurations sampled uniformly from the set of known ones. For each bucket, the agent tracks the recent history of past successes and failures when targeting the corresponding goals (last W = 1800 self-evaluations). C is estimated as the success rate over the most recent half of that history C = C recent . LP is estimated as the difference between the current C (C recent ) and the one evaluated over the first half of the history (C earlier ). This is a crude estimation of the derivative of the C curve w.r.t. time: LP = C recent -C earlier . The sampling probability P i for bucket i is then computed as: .
In addition to the usual LP bias Colas et al. [2019a], this formula favors lower C when LP estimates are fairly similar. The absolute value ensures a regain of interest for buckets that show decreased performances (e.g. forgetting). Implementation details are provided in Supplementary Section 8.
Help from a social partner. After a first period of discovering and learning to reach configurations, agents are helped by sporadic interventions of a social partner. Helping the agents in their exploration, the social partner can organize the scene with non-trivial initial configurations (see details in Supplementary Section 7). This is reminiscent of the dynamics of parent-infant interactions as characterized by the concept of Zone of Proximal Development Vygotsky [1978].

Language Module
We introduce a language-conditioned goal generation module that generates semantic configurations matching the agent's initial configuration and a sentence describing an expected transformation of one object-pair relation. We call this language-conditioned generative model the language module thereafter. This section explains how it is trained and used for language grounding.
Language-conditioned goal generation We implement the language module with a conditional Variational Auto-Encoder (C-VAE) Sohn et al. [2015]. A training dataset is collected via interactions between a trained DECSTR agent and a social partner. For each goal-directed trajectory the agent performs, the social partner provides the description of one of the resulting transformations in the object relations. The set of possible descriptions contains 102 sentences, each describing, in a simplified language, a positive or negative shift for each of the 9 predicates (e.g. get red above green). The C-VAE is similar to the one proposed in Nair et al. [2019]. We simply add an extra condition on the instruction that describes the expected transformation. This way, we add a control on goal generation. This instruction is encoded via a recurrent network that is jointly trained. Supplementary Section 8.4 provides the list of sentences and implementation details.
Language grounding At test time, the agent is instructed to perform a change in one of the object relations by one of the 102 sentences. Conditioned on its current configuration and instruction, it samples a compatible goal from the language module. This module effectively enables agents to ground NL in their internal semantic representations and set of sensorimotor skills. We consider three evaluation settings: 1) performing a single instruction; 2) performing a sequence of instructions; 3) performing a logical combination of instructions. As the agent can generate a set of goals matching any instruction, it can easily combine these sets to perform logical functions of instructions: and is an intersection, or is an union and not is the complement within the set of goals the agent discovered during sensorimotor training. Given sets of compatible goal configurations, agents can also try again: find other target configurations that match the required instruction when previous attempts failed.

Experiments
We first present experiments to evaluate the performance of sensorimotor learning (Phase 1), before considering language grounding (Phase 2). Learning curves report medians and interquartile ranges of the average success rates across goals (SR). These are computed over 5 seeds for baselines and ablations and 10 seeds for DECSTR and its oracle variant with expert-defined buckets. Stars indicate significant differences w.r.t. the DECSTR condition, as reported by Welch's t-tests with α = 0.05 Colas et al. [2019b]. Code and videos can be found at https://sites.google.com/view/decstr/.

Sensorimotor Learning Phase: Discovering and Mastering Semantic Goal Configurations
DECSTR. DECSTR successfully discovers and masters all reachable configurations in its semantic representation space. Figure 2a presents the evolution of SR computed by bucket. The fact that these buckets are learned in increasing order confirms that the time of discovery is a good proxy for difficulty. Figure 2b enters the inner workings of a specific DECSTR agent and reports C, LP and sampling probabilities P as computed online by the agent via self-evaluations. The agent leverages these estimations to guide its goal selection: first focusing on the easy goals from bucket 1, it moves on towards harder and harder buckets as easier ones are mastered (low LP, high C). The Supplementary Section 10 presents more examples of learning trajectories, and dissects the evolution of bucket compositions along training. Baselines. We compare DECSTR to three different baselines. All of them are variants of DECSTR, use the same goal configuration selection based on curriculum learning, and are evaluated with the same metric: matching between goal configuration and final configuration. The Expert Buckets baseline uses an expert similarity-based definition of buckets and, as such, can be considered an oracle w.r.t. the automatic curriculum component of DECSTR. It also leverages a prior knowledge of the set of reachable configurations. The Language Goals and Position Goals baselines do not have access to their current configuration or goal configuration. Instead, the Language Goals baseline uses a jointly learned embedding of a sentence corresponding to that goal configuration. The Position Goals samples a 3D position target for each block so that the resulting semantic configuration matches the semantic goal (see Supplementary Section 9 for extended definitions). Figure 2c shows that DECSTR performs on par with the Expert Buckets condition and dramatically improves over the Language Goals and Position Goals baselines. We hypothesize that the low performance of the Language Goals condition is due to 1) the difficulty to link the continuous current state to the abstract goal expressed through language and 2) by the drifting of the goal representation during training. In the Position Goals condition, we presume low performance is due to the sparsity of the task, as agents need to precisely position each block at their respective targets.
Opportunistic goal reaching. When targeting a given semantic goal configuration, the Position Goals baseline samples particular target positions for each block. This enforces a total distance d pos for the blocks to travel from initial to target configurations. Because it does not use position targets, we hypothesized that DECSTR would find a shorter path d decstr . Computed over 50 episodes for each of the 35 valid goals, we found that ∆ = d pos − d decstr is positive 96% of the time. The average across repetitions is superior to 19 cm for all goals, with a mean of 27 cm (all p < 10 −10 ). This highlights an advantage of using semantic goal configuration over 3D  Ablation studies. Figure 2d presents the results of our ablation studies. Each condition removes one component of DECSTR: 1) Flat replaces our object-centered modular architectures by flat ones; 2) w/o Curr. replaces our automatic curriculum strategy by a uniform goal selection; 3) w/o Sym. does not use the symmetry inductive bias; 4) w/o SP the social partner does not provide non-trivial initial configurations. Only the full version of our architecture enables agents to discover and learn to master all reachable configurations.

Language Grounding Phase
In the sensorimotor learning phase, DECSTR agents acquire a repertoire of abstract skills without access to language. In the language grounding phase, we first train a language-conditioned goal generation module from data collected via interactions between DECSTR and a social partner (Section 3.4). For a given initial configuration and a given sentence, we want the language module to generate all compatible final configurations, and just these.
Language-conditioned goal generation performance. To evaluate the language module, we construct a synthetic, oracle dataset O of triplets (c i , s, C f (c i , s)), where c i is the initial configuration, s is the sentence describing the expected transformation and C f (c i , s) is the set of all final configurations compatible with (c i , s). Note that, on average, C f in O contains 16.7 configurations, while the training dataset D only contains 3.4 (20%). We are interested in two metrics: 1) The Compatibility Probability (CP) is the probability that a goal sampled from the generator belongs to C f ; 2) The Coverage (Cov) is the size of the intersection between C f and the set resulting from sampling the generator 100 times. We compute these metrics on 5 different sets of input pairs (c i , s), each calling for a different type of generalization: 1. Pairs found in D, except pairs removed to form the following test sets. This calls for the extrapolation of known initialization-effect pairs (c i , s) to new final configurations c f (D contains only 20% of C f on average). 2. Pairs that were removed from D, calling for a recombination of known effects s on known c i . 3. Pairs for which the c i was entirely removed from D. This calls for the transfer of known effects s on unknown c i . 4. Pairs for which the s was entirely removed from D. This calls for generalization in the language space, to generalize unknown effects s from related sentences and transpose this to known c i . 5. Pairs for which both the c i and the s were entirely removed from D. This calls for the generalizations 3 and 4 combined.
Our language module demonstrates these 5 types of generalization (see Table 1). Agents can generate goals from situations they never encountered (3). They can generalize the meaning of sentences they never heard (4) and even apply the latter to unknown situations (5). We provide the content of testing sets in Supplementary Section 8.4. We trained the equivalent module to generate continuous target positions for all blocks, which is how the Position Goals condition represents goals. Result show strongly decreased performance, see Supplementary Section 11.  Grounding language in sensorimotor behavior. We investigate how the language module interacts with the sensorimotor skills of the agent. We consider three evaluation settings. In the transition setup, we look at the average success rate of the agent when asked to perform the 102 instructions 5 times each, resetting the environment each time. In the expression setup, we evaluate the agent on 500 randomly generated logical functions of sentences. In both setups, we give the agent 5 attempts, enabling it to resample new compatible goals when the previous failed (without reset). Success rates after 1 (SR1) and 5 (SR5) attempts are reported in Table 2. In the sequence setup, we ask the agent to execute 20 random sequences of instructions without reset and report the average number of successes before the agent fails: L = 14.9 ± 5.7 (std). Our 10 DECSTR agents are evaluated with the 10 C-VAE models evaluated above. These results show that the language module efficiently implements language grounding. Agents achieve instructed transitions almost all the time, resampling alternative goals when previous ones failed. They only fail when a previous trajectory kicked the blocks out of reach.

Discussion
This paper introduces DECSTR, a learning architecture that discovers and masters all reachable configurations from a set of relational spatial primitives, before undertaking an efficient language grounding phase. This was made possible by the use of object-centered inductive biases, a new form of automatic curriculum learning and a novel language-conditioned goal generation module.
Semantic representations. In contrast with specific target coordinates, semantic configurations allow to specify abstract goals which can be achieved in an opportunistic way and better correspond to human-like language descriptions. This favors the grounding of language into internal sensorimotor representations. This pivotal semantic space also allows to decouple sensorimotor learning from language grounding, a sequentiality observed in infants Piaget [1977]. This paper investigates one example of such predicate-based representations, using spatial relations that infants are known to master early Mandler [2012]. We leave for future work the problem of learning other representations from data.
A new approach to language grounding. Our language grounding approach is the first to enable a decoupling between sensorimotor learning and language grounding. In addition, it fosters a diversity of possible behaviors for any given instruction. Indeed, while an instruction-following agent trained on goals like put red close_to green would just push the red block towards the green one, our agent can generate many matching goal configurations. It could build a pyramid, make a blue-green-red pile or target a dozen other compatible configurations. This enables agents to try again, to find alternative approaches to satisfy a same instruction when first attempts failed. Our goal generation module can also generalize to new sentences or transpose instructed transformations to unknown initial configurations. Finally, the goal generation module automatically enables agents to deal with any logical expression made of instructions by combining generated goal sets. Note that nothing in the architecture prevents language from being grounded during sensorimotor learning, which would result in "overlapping waves" of sensorimotor and linguistic development Siegler [1998].
Semantic configurations of variable size. Considering a constant number of blocks and, thus, fixed-size configuration spaces is a current limit of our approach. We could imagine scaling to inputs of variable sizes by leveraging Graph Neural Networks as in Li et al. [2019]. Corresponding semantic configurations could be represented as a set of vectors, each encoding information about a predicate and the objects it applies to. These representations could be handled by Deep Sets Zaheer et al. [2017]. This would allow to target partial sets of predicates that would not need to characterize all relations between all objects, thus preventing the curse of dimensionality.
Continuous goals and binary goals The Position Goals baseline is close to the algorithms presented in Lanier et al. [2019] and Li et al. [2019]. The former relies on 3 key ingredients: a heavy curriculum on initializations, a curiosity mechanism and a custom curriculum involving object-wise hindsight replay while the latter relies on a strong curriculum on the number of blocks in the scene. Without these mechanisms, using target block positions does not perform well (Section 9). Based on these results, we argue that using abstract goal configurations helps reducing the complexity of such tasks. As a result, DECSTR does not need as many additional mechanisms to be successful.
Conclusion. Our work is a first step towards artificial agents endowed with higher-level cognitive capabilities such as reasoning about their own goals Eppe et al. [2019] or leveraging natural communication with a caregiver about these goals. Here, the role of the caregiver was limited to: 1) helping the agent to experience non-trivial configurations and 2) describing the agent's behavior in a simplified language. In the future, we intend to leverage social interactions to guide agents in the design of their own semantic representations via the use of demonstrations, goal emulations etc.

Supplementary Material
This supplementary material includes a more formal definition of our semantic configurations (Section 6) and provides details about the Fetch Manipulate environment (Section 7), the DECSTR architecture (Section 8) and the considered baselines (Section 9). It also presents additional results about the learning trajectories of DECSTR agents (Section 10) and the performance of a continuous language module (Section 11). Hyperparameters used in the various components of DECSTR are listed in Section 12. Our code will be accessible at https: //sites.google.com/view/decstr/.

Formal Definition of Semantic Configurations
Semantic configurations are based on a collection of formal systems known as predicate logic or first-order logic. They use k-ary relations to describe possible connections between k quantified variables. This paper focuses on spatial binary predicates characterizing spatial relations between pairs of physical objects. We provide formal definitions, properties and examples below.
Binary predicates Consider a finite set of objects O = {o1, o2, ..., oM }. A binary predicate p associated with a semantic relation r is an expression that takes as input any ordered pair of objects (oi, oj) ∈ O 2 . p(oi, oj) is said true if and only if "oi r oj" is verified. For simplicity, we refer to p and r interchangeably.
Examples of binary predicates. We consider the objects o1 and o2.
Semantic mapping functions. To achieve symbol grounding into non-symbolic sensorimotor interactions using predicates, we define a semantic mapping function f associated with the binary predicate p as the probability that p is true given the states of the considered objects. Formally, if we consider the objects oi, oj and their respective states si, sj, then: f (si, sj) = P (p(oi, oj) | si, sj).
This paper assumes oracle deterministic semantic mapping functions, i.e. f is a Boolean function in {0, 1}. Practically, we hard-code a function, assumed internal to the agent, that uses predefined fixed thresholds to determine whether a predicate is true or false given the states of the considered objects. For example, for the close predicate, it outputs 1 if and only if the Euclidean distance between the two considered objects is below a defined threshold. For the sake of simplicity, we omit the word deterministic.
Symmetry and asymmetry. Consider a finite set of objects O = {o1, o2, ..., oM } and a binary predicate p. The predicate p is said to be symmetric if and only if, for any ordered pair of objects (oi, oj) ∈ O 2 , "oi r oj" and "oj r oi" are equivalent. As a result, the corresponding semantic mapping function f needs to be symmetric, i.e. f (oi, oj) = f (oj, oi). The predicate p is said to be asymmetric iff, for any ordered pair (oi, oj) ∈ O 2 , "oi r oj" implies not "oj r oi".
Examples. We consider the objects o1 and o2.
• close is symmetric: "o1 is close to o2" ⇔ "o2 is close to o1". The corresponding semantic mapping function is based on the Euclidean distance, which is symmetric.
• above is asymmetric: "o1 is above o2" ⇒ not "o2 is above o1". The corresponding semantic mapping function evaluates the sign of the difference of the object Z-axis coordinates.
Effective number of predicate relations. Semantic configurations based on spatial relations. Let (pi) i∈[1.
.P ] be a list of P binary predicates.
The concatenation of the evaluations of the semantic mapping functions fi on the Kpi pairs of objects forms a semantic configuration. It is an abstract representation of a scene which characterizes all relations defined by the (pi) predicates among the M objects. This defines a binary semantic configuration space Cp = {0, 1} Kc , where Kc = P i=1 Kp i . If any world configuration can be mapped to Cp, not all configurations are reachable (e.g. o1 cannot be above and below o2 at the same time).
7 Fetch Manipulate environment and Problem Statement.
Semantic representation space in Fetch Manipulate. In this paper, we restrict the semantic representations to the use of the close and above binary predicates applied to M = 3 objects. The resulting semantic configurations are formed by: a(o1, o2), a(o2, o1), a(o1, o3), a(o3, o1), a(o2, o3), a(o3, o2)], where c() and a() refer to the close and above predicates respectively and (o1, o2, o3) are the red, green and blue blocks respectively. Figure 3 displays visual representations of some example configurations. Helping exploration with biased initialization. A DECSTR agent discovers configurations via its own exploration and can only target known configurations. As it learns to master easier configurations, its behavior becomes less exploratory and more focused, which decreases the probability to discover new complex configurations. Inspired by the framework of Zone of Proximal Development that describes how parents organize the learning environment of their children Vygotsky [1978], we allow a social partner to set up the learning environment of DECSTR agents. The social partner -acting as a caregiver -regularly organizes blocks in non-trivial configurations containing at least one stack at the start of learning episodes. It initializes blocks in non-trivial configurations with probability pnon-trivial. These non-trivial configurations are split into stacks of 2 with probability p2 and stacks of 3 with probability p3. In addition, a block is initially put in the agent's gripper with probability pgrasp. Note that these initializations are not used during offline evaluations. Table 3 presents the corresponding hyperparameters. The DECSTR algorithm can be split in two independent phases. Phase 1 is a goal-directed sensorimotor learning phase. Once DECSTR has explored and mastered all reachable configurations in its semantic representation space, Phase 2 starts. Phase 2 is about language grounding: learning a mapping between language and internal representations of semantic configurations and associated sensorimotor skills. Algorithm 1 presents the pseudocode. The sensorimotor learning phase alternates between two steps: data acquisition and internal model updates.
Data acquisition. A DECSTR agent has no prior on the set of reachable semantic configurations. Its first goal is sampled uniformly from the semantic configuration space. Now that it has a target, the agent can start interacting with its environment. This interaction generates data: a trajectory of sensory states s, actions a and configurations cp. The last configuration c T p achieved in the episode after T time steps is considered stable and is added to the set of reachable configurations. As it interacts with the environment, the agent explores the configuration space, discovers which configurations are reachable and selects new targets.
Internal models updates. A DECSTR agent updates two models: its curriculum strategy and its policy. The curriculum strategy can be seen as an active goal sampler. It biases the selection of goals to target and goals to learn about. The policy is the module controlling the agent's behavior and is updated via Reinforcement Learning.
Interaction with a social partner to collect a Dataset 28: language_module.train(dataset)

Object-Centered Inductive Biases
Object-centered architectures. In the proposed Fetch Manipulate environment, the three blocks share the same set of attributes (position, velocity, color identifier). Thus, it is natural to encode a relational inductive bias in our architecture. The behavior with respect to a pair of objects should be independent from the position of the objects in the inputs. This inductive bias is enforced by architectures inspired from Deep Sets Zaheer et al. [2017]. Colas et al. [2020] first proposed an object-based modular architecture for RL, that was subsequently extended and studied in more depth in Karch et al. [2020]. Here we propose policy and critic architectures similar to the one used in Karch et al. [2020] for reward prediction. The architecture used for the policy is depicted in Figure 4. A shared network (N Nshared) encodes the concatenation of: 1) agent's body features; 2) object pair features; 3) current configuration (cp) and 4) current goal g. This is done independently for all object pairs. No matter the location of the features of the object pair in the initial observations, this shared network ensures that the same behavior will be performed: skills are transferred between object pairs. A sum is then used to aggregate the outputs of these encodings, before a final network (N Npolicy) maps the aggregation to actions a. The critic follows the same architecture, where a final network N Ncritic maps the aggregation to an action-value Q. Parallel encodings of each pair-specific inputs can be seen as different modules trying to reach the goal by only seeing these pair-specific inputs. The intuition is that modules dealing with the pair that should be acted upon to reach the goal will supersede others in the sum aggregation. Combinations, permutations and the symmetry inductive bias. Although our architecture could theoretically work with combinations of objects (3 modules), we found permutations to work better in practice (6 modules). With combinations, the shared network would need to learn to put block A on block B to achieve a predicate above(oi, oj), and would need to learn the reverse behavior (put B on A) to achieve the symmetric predicate above(oj, oi). With permutations, the shared network can simply learn one of these behaviors (e.g. A on B). Considering the predicate above(oA, oB), at least of of the modules will have objects organized so that this behavior is the good one: if the permutation (oB, oA) is not the right one, permutation (oA, oB) will be. The symmetry bias is explained in Main Section 3.2. It leverages the symmetry of the behaviors required to achieve the predicates above(oi, oj) and above(oj, oi). As a result, our two goal configurations are: g1 = [c(o1, o2), c(o1, o3), c(o2, o3), a(o1, o2), a(o1, o3), a(o2, o3)], g2 = [c(o1, o2), c(o1, o3), c(o2, o3), a(o2, o1), a(o3, o1), a(o3, o2)]. g1 is used in association with object permutations (oi, oj) with i < j and g2 is used in association with object permutations (oj, oi) with i < j. As a result, the shared network automatically ensures transfer between predicates based on symmetric behaviors.

Policy Updates with a Goal-directed Soft Actor-Critic.
We want DECSTR agents to explore a semantic configuration space and master the corresponding reachable configurations. We frame this problem in a goal-conditioned MDP framework Schaul et al. [2015]: M = (S, Gp, A, T , R, γ). The state space S is the usual sensory space augmented with the configuration space Cp. The goal space Gp is equal to the configuration space Gp = Cp. A is the action space, T : S × A × S → [0, 1] is the unknown transition probability, R : S × A → {0, 1} is a sparse reward function and γ ∈ [0, 1] is the discount factor.
We conduct policy updates with Soft Actor-Critic (SAC) Haarnoja et al. [2018], a state-of-the-art off-policy actor-critic algorithm. We also use the Hindsight Experience Replay (HER) Andrychowicz et al. [2017]. This mechanism enables agents to learn from failures by reinterpreting past trajectories in the light of different goals from the ones originally targeted. HER was designed for continuous goal spaces, but can be directly transposed for discrete goals Colas et al. [2019a]. In our setting, we simply replace the originally targeted goal configuration by the currently achieved configuration in the transitions fed to SAC. Here also, we use our automatic curriculum strategy. The LP-C-based probabilities are used to sample goals to learn about. When a goal g is sampled, we search the experience buffer for the collection of episodes that ended in the configuration cp = g. From these episodes, we sample a transition uniformly. The HER mechanism substitutes the original goal with one of the configurations achieved later in the trajectory. This substitute g has high chances of being the sampled one. It is, at least, a configuration on the path towards it, as it is sampled from a trajectory leading to it. The hindsight mechanism is thus biased towards the goals sampled by the agent.

Language-Conditioned Goal Generator Training
We use a conditional Variational Auto-Encoder (C-VAE) Sohn et al. [2015]. Conditioned on the initial configuration and a sentence describing the expected transformation of one object relation, it generates compatible goal configurations. After the first phase of goal-directed sensorimotor training, the agent interacts with a hard-coded social partner as described in Main Section 3.4. From these interactions, we obtain a dataset of 5000 triplets: initial configuration, final configuration and sentence describing one change of predicate from the initial to the final configuration. The list of sentences used by the synthetic social partner are provided in Table 4. Note that red, green and blue refer to objects o1, o2, o3 respectively. Content of test sets. We describe the 5 test sets: 1. Test set 1 is made of input pairs (ci, s) from the training set, but tests the coverage of all compatible final configurations C f , 80% of which are not found in the training set. In that sense, it is partly a test set. Testing on logical expressions of instructions. To evaluate our agents on logical functions of instructions, we generate three types of expressions: 1. 100 instructions of the form "A and B" where A and B are basic instructions corresponding to shifts of the form above 0 → 1 (see Table 4). These intersections correspond to stacks of 3 or pyramids. Here also, any instruction can be replaced by its negation with probability 0.5.

Sensorimotor Learning Baselines
Expert Buckets Baseline. In the Expert Buckets baseline, we replace the automatic bucket generation of DECSTR with an expert-predefined set of buckets. These expert buckets group goals based on human-defined a priori measures of similarity and difficulty. It assumes a prior knowledge of the set of reachable configurations, which are ruled out. The 5 predefined buckets are constructed as follows: • Bucket 1 contains all configurations characterized by a single close relation between a pair of objects and no above relations (4 configurations).
• Bucket 2 contains all configurations with 2 or 3 close relations and no above relations (4 configurations).
• Bucket 3 contains configurations with 1 stack of 2 blocks and a third block that is either away or close to the base, but is not close to the top of the stack (12 configurations).
• Bucket 4 contains configurations with 1 stack of 2 blocks and the third block close to the stack, as well as pyramid configurations (9 configurations).
These buckets are the only difference between the Expert Buckets baseline and the DECSTR algorithm.
Language Goals Baseline. For the Language Goals baseline, we define a set of 35 language goals, one for each valid semantic goal configuration (see Table 5). This baseline is highly similar to the DECSTR algorithm. It samples binary goal configurations and receives rewards when its current semantic configuration matches it. There are two differences: 1) the policy is not conditioned neither on the agent's current semantic configuration nor on the semantic goal configuration but on an encoding of the corresponding language sentence; 2) As the networks are not conditioned on semantic goal configurations, we do not apply the symmetry inductive bias. The language encoding is the hidden state of a recurrent neural network trained jointly with the policy (one hidden layer, tanh units, hidden size 100). This baseline tests the capacity of language to express abstract goals. Note that our semantic configurations are simple enough to be represented by one sentence, which might not be the case with more complex semantic representations. Note also, that this baseline assumes hidden semantic representations and use them to sample semantic goals and to compute rewards. . Hence the goal vector size is 9. Here again, the algorithm samples semantic goal configurations using our automatic curriculum learning. However, it uses a module to convert these into specific randomly-sampled target positions, see Figure 5. The agent is not conditioned on its current semantic configuration or semantic goal configuration. For this reason, we do not apply the symmetry bias. The binary reward is positive when the maximal distance between a block and its target is below 5 cm, that is the size of a block. All other components match the DECSTR algorithm. Note that, although we use a different reward function, success rates are still reported using matching between target and current semantic configurations to ensure fairness with other baselines and DECSTR.

DECSTR Learning Trajectories
This sections delves into the inner workings of the DECSTR algorithm.
Automatic bucket generation. Figure 6 represents the evolution of the content of buckets during training (epochs 1, 50 and 100). Each pie chart corresponds to a reachable configuration and represents the distribution of configurations into buckets across 10 different seeds. Blue, orange, green, yellow, purple represent buckets 1 to 5 respectively and grey are undiscovered configurations. Recall that, at each moment, the discovered configurations are equally spread over the 5 buckets. The bucket attribution of a given configuration may thus change as new configurations are discovered, so that the ones discovered earlier are assigned buckets with lower indexes. Goals are organized by their bucket assignments in the Expert Buckets condition (top-bottom organization).
After the first epoch (left), DECSTR has discovered all configurations from the expert buckets 1 and 2, and some runs have discovered a few other configurations. After 50 epochs, more configurations have been discovered but they are not always the same across runs. Finally, after 100 epochs, all configurations have been discovered.
Buckets have then reached their final organization and we can compare their contents with expert-defined buckets. It seems that easier goals (top-most group) are discovered first and assigned in the first-easy buckets (blue and orange). Hardest configurations (stacks of 3, bottom-most group) seem to be discovered last and assigned the last-hardest bucket (purple). In between, different runs show different compositions, which are not always aligned with expert-defined buckets. Goals from expert-defined buckets 3 and 4 (third and fourth group from the top) seem to be attributed different automatic buckets in different runs. This means that they are discovered in different orders depending on the runs. In summary, easier and harder goals from expert buckets 1 -2 and 5 respectively seem to be well detected by our automatic bucket generations. Goals in medium-level expected difficulty as defined by expert buckets seem not to show any significant difference in difficulty for our agents.
Learning trajectories. Figure 7 shows the evolution of internal estimations of the competence C, the learning progress LP and the associated sampling probabilities P. Note that these metrics are computed by DECSTR agents online, as they self-evaluate on random discovered configurations. Learning trajectories seem to be uniform across different runs, and buckets are learned in increasing order. This support our hypothesis stating that the time of discovery is a good proxy for goal difficulty. In that case, configurations discovered first end up in the lower index buckets and are indeed learned first. Note that a failing automatic bucket generation would assign goals to random buckets. This would result in uniform measures of learning progress across different buckets, which would be equivalent to uniform goal sampling. As Main Figure 2d shows, DECSTR performs much better than the random goals conditions. This proves that our automatic bucket algorithm generates useful goal repartitions.

Language-Conditioned Continuous Goal Generation
In this section, we look at the performance of our language-conditioned goal generator when it handles continuous position goals instead of semantic goal configuration. This type of language module could be used in combination with algorithms like our Position Goals baseline, or approaches like Lanier et al. [2019], Li et al. [2019]. It is trained to generate targets for each of the three blocks, as a function of the initial block positions and a sentence describing the expected transformation. When a continuous goal is generated, we encode it in the corresponding configuration using our semantic mapping function, and see whether the transformation from the initial configuration to the final one respects the instruction. Because the states are continuous, we replace the binary cross entropy loss by a mean squared error. We use the exact same training and testing sets as the semantic  Figure 6: Evolution of the content of buckets as organized by our automatic bucket generation: epoch 1 (2400 episodes, left), 50 (middle) and 100 (right). Each pie chart corresponds to one of the 35 valid configurations. It represents the distribution of the bucket attributions of that configuration across 10 runs. Blue, orange, green, yellow, purple represent automatically generated buckets 1 to 5 respectively (increasing order of difficulty) and grey represents undiscovered configurations. Goals are organized according to their expert bucket attributions in the Expert Goals condition (top-bottom organization). goal generator. Table 6 provides the Cov and CP metrics. Most of the time, the trained generator samples continuous goals corresponding to 1 or 2 final configurations, while the set of possible final configurations contains 16.7 configurations on average. This results in a very low coverage of possible final configurations. However, it seems that the few generated continuous states mostly correspond to compatible configurations in the 2 first test sets (good Compatibility Probability). These performance are far below the ones of our language module based on semantic configurations. These results show an additional advantage of using semantic configurations: it facilitates language grounding.

Hyperparameters and Implementation Details
Parallel implementation of SAC-HER. We use a parallel implementation of the Soft Actor-Critic algorithm (SAC) Haarnoja et al. [2018] with an Hindsight Experience Replay (HER) Andrychowicz et al. [2017] mechanism. We use 24 workers in parallel, where each worker maintains its own replay buffer of size 10 6 and performs its own updates. Updates are summed over the 24 actors and the updated network are broadcast to all workers. Each worker alternates between 2 episodes of data collection and 30 updates with batch size 256. To form an epoch, this cycle is repeated 50 times and followed by the offline evaluation of the agent on each reachable goal. An epoch is thus made of 50 × 2 × 24 = 2400 episodes.
Goal sampler updates. The agent performs self-evaluations with probability self _eval = 0.1. During these runs, the agent targets uniformly sampled discovered configurations and exploration noise applied on actions is disabled. This enables the agent to self-evaluate on each goal. Goals are organized into buckets. Main Section 3.3 presents our automatic bucket generation mechanism. Once buckets are formed, we compute C, LP and P as detailed in Main Section 3.3, based on windows of the past W = 1800 self-evaluation interactions for each bucket.
Modular architectures. The shared network of our modular architecture N Nshared is a 1-hidden layer network of hidden size 256. After all pair-specific inputs have been encoded through this module, their output (of size 84) are summed. The sum is then passed through a final network with a hidden layer of size 256 to compute the final actions (policy) or action-values (critic). All networks use ReLU activations, and are initialized with the Xavier initialization. We use Adam optimizers, with learning rates 10 −3 . More details about our learning hyperparameters are given in Table 7.
Language-conditioned goal generator. The encoder is a fully-connected neural network with two layers of size 128 and ReLU activations. It takes as input the concatenation of the final binary configuration and its two conditions: the initial binary configuration and an embedding of the NL sentence. The NL sentence is embedded with an recurrent network with embedding size 100, tanh non-linearities and biases. The encoder outputs the mean and log-variance of the latent distribution of size 27. The decoder is also a fully-connected network with two hidden layers of size 128 and ReLU activations. It takes as input the latent code z and the same conditions as the encoder. As it generates binary vectors, the last layer uses sigmoid activations. We train the architecture with a mixture of Kullback-Leibler divergence loss (KDloss) w.r.t a standard Gaussian prior and a binary Cross-Entropy loss (BCEloss). The combined loss is loss = BCEloss + β × KDloss with β = 0.6. We use an Adam optimizer with a learning rate of 5 × 10 −4 , a batch size of 128 and optimize for 150 epochs. As training is fast (≈ 2 min on a single cpu), we conducted a quick hyperparameter search over β, layer sizes, learning rates and latent sizes (see Table 8). We found robust results for various layer sizes, various β below 1. and latent sizes above 9.
Computing resources. The sensorimotor learning experiments contain 8 conditions: 2 of 10 seeds and 6 of 5 seeeds. Each run leverages 24 cpus (24 actors) for about 72h for a total of 9.8 cpu years. Experiments presented in this paper requires machines with at least 24 cpu cores. The language grounding phase runs on a single cpu and trains in a few minutes.