{'fraction_non_alphanumeric': 0.03053456636968535, 'fraction_numerical': 0.01215179880455622, 'mean_word_length': 4.997463645586743, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning, Planning, and Control in a Monolithic Neural Event Inference Architecture

We introduce a dynamic artificial neural network-based (ANN) adaptive inference process, which learns temporal predictive models of dynamical systems. We term the process REPRISE, a REtrospective and PRospective Inference SchEme. REPRISE infers the unobservable contextual state that best explains its recently encountered sensorimotor experiences as well as accompanying, context-dependent temporal predictive models retrospectively. Meanwhile, it executes prospective inference, optimizing upcoming motor activities in a goal-directed manner. In a first implementation, a recurrent neural network (RNN) is trained to learn a temporal forward model, which predicts the sensorimotor contingencies of different simulated dynamic vehicles. The RNN is augmented with contextual neurons, which enable the compact encoding of distinct, but related sensorimotor dynamics. We show that REPRISE is able to concurrently learn to separate and approximate the encountered sensorimotor dynamics. Moreover, we show that REPRISE can exploit the learned model to induce goal-directed, model-predictive control, that is, approximate active inference: Given a goal state, the system imagines a motor command sequence optimizing it with the prospective objective to minimize the distance to a given goal. Meanwhile, the system evaluates the encountered sensorimotor contingencies retrospectively, adapting its neural hidden states for maintaining model coherence. The RNN activities thus continuously imagine the upcoming future and reflect on the recent past, optimizing both, hidden state and motor activities. In conclusion, the combination of temporal predictive structures with modulatory, generative encodings offers a way to develop compact event codes, which selectively activate particular types of sensorimotor event-specific dynamics.


Introduction
The predictive brain perspective and active inference principles have strongly influenced cognitive science over the last years (Bar, 2009;Butz and Kutter, 2017;Clark, 2016;Friston, 2009;Hohwy, 2013). Although predictive encodings have been first shown to yield promising results in artificial neural networks focusing on vision Ballard, 1998, 1999), it remains highly challenging to realize these principles in scalable, temporal dynamic artificial neural network models, and particularly models that enable flexible, goal-directed planning (but see Najnin and Banerjee, 2017 for a recent, promising approach in phonological speech production). Moreover, it remains unclear how abstracted, hierarchical structures may be developed effectively (Botvinick and Weinstein, 2014;McClelland et al., 2010) -structures that are believed to be essential for enabling the generation of flexible, adaptive goal-directed behavior by means of hierarchical, model-based planning and reinforcement learning (Botvinick et al., 2009).
Despite the recent remarkable successes in playing somewhat challenging computer games and the board game GO (Mnih et al., 2015;Silver et al., 2016), neural networks generally still seem to lack a deeper understanding of the underlying problem domain. As a result, the developed systems are rather inflexible, for example, when multiple, different tasks need to be solved by the same architecture or when the reward function changes. There are certainly strategies that can help -such as more effective episodic replay or task-specific weight and neural manipulations (Kirkpatrick et al., 2016). Nonetheless, deep and recurrent neural networks do not yet learn and think with the flexibility of humans (Lake et al., 2017).
Here, we introduce a novel retrospective and prospective temporal inference scheme (REPRISE) for recurrent artificial neural networks (RNNs). REPRISE combines weight adaptation (i.e. model inference) with neural activity adaptation (i.e. contextual hidden state inference) and model-predictive, anticipatory control and behavior (i.e. active inference). As a result, REPRISE does not optimize a static reward function, but it can flexibly plan context-and task-dependently. Moreover, REPRISE offers a first step towards the development of hierarchical hidden, generative structures, which can be closely related to the concept of event cognition.
The event cognition principle comes from cognitive psychology. It was shown that humans have a strong tendency to segment a continuous sensorimotor stream into meaningful events and eventtransitions, leading to the proposal of an event segmentation theory (EST) (Radvansky and Zacks, 2014;Zacks and Tversky, 2001;Zacks et al., 2007). Concurrently, the theory of event coding (TEC) has proposed integrative action-effect codes, referring to them as event codes (Hommel et al., 2001). Similarly, forward-inverse control schemes have been put forward as models of human behavior (Wolpert and Kawato, 1998;Wolpert and Flanagan, 2016), where the involved forward-inverse models essentially encode interaction events. Even the memorization of experienced episodes appears eventsegmented and event-focused (Richmond et al., 2017). Moreover, memorized events can be used not only for processing current sensorimotor information, but also for reflecting on the past and for imagining potential futures (Schacter et al., 2012). Combined with the predictive coding perspective on cognition, our mind appears to have the tendency to cluster sensorimotor contingencies into predictive events (Butz, 2016), continuously adapting to the current event-respective circumstances or to imagined past, future, or even fully hypothetical events (Bar, 2009;Buckner and Carroll, 2007).
REPRISE offers a first neural implementation that shows the emergent tendency to cluster different types of predictively encoded sensorimotor dynamics into compact event codes. While learning, the algorithm infers stable hidden states, which tend to contrast distinct sensorimotor dynamics. When activating active, prospective inference, the system is able to adapt its internal state and its motor behavior in such a way that context-adaptive goal-directed control becomes possible. From a control perspective, the system can be said to approximate model-predictive control routines, where the model is learned by and then encoded in an RNN.
As a first evaluation scenario, we train REPRISE to control three different types of "vehicles" in a simple but dynamic 2D simulated environment. Crucially, we show that REPRISE is able to distinguish the three types of vehicles and control them effectively in a goal-directed manner even when no information about the vehicle identity or even the fact that there are three different vehicles is provided. We believe that this method may be very well-suited to learn event-oriented abstractions and event hierarchies, but future work is necessary to scale the system and apply it to more challenging scenarios.

System Architecture and Inference Mechanisms
We now detail the mechanisms implemented by our retrospective and prospective temporal inference scheme (REPRISE), which is implemented in a recurrent artificial neural network (RNN). REPRISE infers retrospectively the unobservable current event context (here the controlled vehicle), which best explains the recent sensorimotor experiences, while it concurrently infers motor control commands prospectively in a goal-directed manner. We thereby build on our previous work, which had accomplished prospective, active motor control inference (Otte et al., 2017a,b) but not retrospective inference. Our results suggest that REPRISE can learn and apply both, effective goal-directed control and event-oriented, system state inference.
In order to introduce REPRISE, we distinguish between the actual (not directly observable) dynamical system φ and the model Φ of this system, which is encoded by an RNN. Focusing on a discrete-time dynamical system, at a certain point in time t, the (not directly observable) current state of the dynamical system φ may be denoted by ϑ t , such that the progression through time is determined by

Temporal Forward Model
The model Φ is trained to approximate these dynamics, inferring its parameters from sensorimotor experiences during learning. However, seeing that we are dealing with a dynamic, partially observable Markov decision process (POMDP) (Sutton and Barto, 1998), the true system state ϑ t is typically not directly deducible from current observables s t ∈ R n . Thus, the dynamical system's internal state σ t must be inferred in each iteration from the current observables s t , the current motor activities denoted by x t ∈ R k , and the previous system state estimate σ t−1 . With the help of the system's model Φ, the next system state σ t+1 and the consequent sensory expectationss t+1 are determined by where the mapping Φ essentially models the temporal forward dynamics of the system. Thus, the next system state and sensory expectations depend on the current sensor (s t ) and motor control (x t ) activities as well as, in principle, on the entire state history, which is encoded in the (hidden) state components (σ t ) in compressed form. While learning the model, that is, while pursuing model inference, the system attempts to minimize the squared loss between predicted and encountered sensory information over time, that is, summing the accumulated losses over the gathered experiences {s 1 , . . . , s T }.

Multiple Dynamical Systems
In this paper we consider not only a single dynamical system, but an ensemble of multiple dynamical systems φ = {φ 1 , . . . , φ u }, which cause different sensorimotor contingencies over time. These systems differ from each other concerning their behavior, but share the same input, state, and output dimensions. During model inference, the model Φ is trained to approximate all of these dynamical systems within one monolithic RNN architecture. As a result, the challenge is to approximate the particular dynamical system φ i that is currently active, given observed state s t and control commands x t , in order to be able to accurately predict future sensorimotor dynamics and thus to control the dynamical system itself in an anticipatory, goal-directed manner.
When evaluating REPRISE we will first provide the identity of the currently active dynamical system φ i in an additional context input neurons c ∈ R u , which is simply added as additional input and which is initially encoded as a one-hot vector (i-th component is set to 1, rest to 0). This encoding is closely related to parametric bias neurons, which can be viewed as an indicator of the current event the system is situated in Sugita et al. (2011);Tani (2017). During goal-directed control, however, we will infer the values of this vector online. Later on, we will infer this vector also during training (never providing information about the current vehicle identity) and will show that the different vehicles tend to be encoded separately in the provided contextual neurons.

REPRISE
Given an imagined action sequence, an initial state, and the identity of the current dynamical system, the RNN can predict a state progression that is expected when executing the action sequence by Figure 1: Illustration of REPRISE for two consecutive time steps t (top part) and t + 1 (bottom part). Note that there is only one RNN, whose activities are buffered. The right (green shaded) boxes illustrate future imaginations actively inferring prospective motor activities, while the left boxes (gray shaded) show retrospections about the recent past for system state inference (including event state c t and hidden state σ t ). Black lines indicate context and information forward flow, while the red lines indicate gradient flow.x t andc t refer to the action and context input vectors, respectively, for a particular time step t .s t τ refers to a particular sensory prediction in the τ -th optimization cycle, whereas s t refers to a desired sensory goal state. means of the learned temporal forward model Φ. To effectively control the system, however, the inverse mapping is required, that is, an action sequence needs to be inferred to approach a desired goal-state (or follow a sequence of goal-states) from an initial state. This becomes even more difficult when the identity of the current actual dynamical system φ i is unknown and has to be inferred as well. In this section we introduce the REPRISE algorithm -a concurrent retrospective and prospective inference scheme, which solves the twofold system identification and goal-directed control problem. Figure 1 shows the dynamic processes REPRISE unfolds for two consecutive time steps. During each step, both a retrospective and a prospective inference phase is executed.
In the retrospective phase, the gradient is propagated R time steps into the past, to reflect on the states that were just experienced. The gradient is fed by the discrepancy between previously predicted system statess t−i , with i ∈ 0, . . . , R, and the actually observed system states s t−i , minimizing the quadratic loss over this time horizon (3). The discrepancy is then mapped onto the assumed context input c t−i -essentially a sub-vector of s t−i -indicating the dynamical system φ i that is presumably currently active. Because REPRISE is designed to distinguish different events, c is set to a constant value of the time horizon R -essentially summing up the gradient signals received by c t−1 . . . c t−R when applying gradient descent. Additionally, the error gradient can be used to adapt the RNN's hidden state at time step t − R − 1, that is, σ t−R−1 , such that it better fits the changing context input. As a result, the RNN avoids disadvantageous or even undefined sensory input, motor command, hidden state combinations. After neural activity adaptation via gradient descent (we later contrast standard gradient descent plus momentum term with Adam, Kingma and Ba, 2014), the neural activities are propagated forward again to the present time step, with respect to the inferred hidden state and context input, and the already recorded motor commands and observed system states, yielding an updated σ t . This retrospective neural activity inference cycle may be executed r times.
Additionally, retrospective weight adaptation is applied during model learning, again focusing on minimizing the quadratic loss (3). This essentially corresponds to standard back-propagation through time learning. Note that we usually use a deeper retrospective time horizon R w for model learning and a smaller learning rate η. Otherwise the RNN would behave more like an adaptive filter as it would not learn the (versatile) model characteristics but rather over-fit the recent signal shape.
In the prospective phase, neural activities are projected P time steps into the future, starting with the inferred current internal system state σ t and hypothetically executing a sequence of motor commands x t+i , which was inferred previously. The discrepancies between the predicted futures t+i and desired goal state sequences s t+i , with i ∈ 1, . . . , P , are then propagated backwards through time from the imagined future back to the present time step, while the gradient is projected onto the individual neurally encoded anticipated motor activity sequencex t+i , effectively optimizing it in the light of the current system state estimates and the desired goal state. This prospective inference cycle is executed p times.
After the retro-and prospective inference phases, the inferred motor activity x t is executed by the system φ and the forward RNN is updated via (2). This closes the processing loop, repeating REPRISE in the following time step (t + 1).

System Evaluations
Our experiments are based on a two dimensional dynamical system simulation with u = 3 types of "vehicles", constituting three dynamical systems: φ 1 is a multi-copter-like vehicle, which we call rocket, φ 2 is a static omnidirectional vehicle, which we call stepper, and φ 3 is a dynamical, omnidirectional gliding vehicle, which we call glider. The rocket is influenced by simulated gravity and undergoes inertia. It has two propulsion motors that are spread at a 45 • angle from the vertical axis on both sides, inducing thrust forces in the respective direction. The two other motor inputs are ignored in the case of the rocket. The stepper has four thrust motors that are spread at 45 • and 135 • angle from the vertical axis to both sides, inducing steps in the opposite direction. Finally, the glider has the same four thrust motors as the stepper. However, in contrast to the stepper, the glider undergoes inertia without any friction. Each motor unit can be throttled within the interval [0, 1]. Upon invocation, each vehicle is positioned in a rectangular free space of size 3 × 2 units. It is surrounded by borders, which block the vehicle.
All presented evaluations below are based on ten independently trained networks, averaging the achieved results. In this section, we focus on evaluating REPRISE when the context neuron activities are set during training to distinct one-hot vector values for the different vehicles. In the subsequent section, we then investigate learning and performance when context values are inferred during learning as well.

Model Learning
During training, stochastic back-propagation through time optimized the weights of the considered RNN architectures based on simulated sensorimotor experiences, learning in a self-supervised manner. Experiences were generated by executing pseudo-random motor commands x ∈ [0, 1] 4 , where motor command generation was such that sufficient upwards thrust was generated and a reasonable exploration of the complete rectangular free space was loosely ensured.
At each time step the network is fed with the current position of the vehicle (s ∈ {[−1.5, 1.5], [0, 2]}), the current four motor commands (activities of the four thrust motors as forces; for the rocket, the second two motor values have no effect; mass of vehicles is set to .1), and a three bit one-hot vector, which indicates the vehicle that is currently controlled, i.e., which φ i applies. The network output is the prediction of vehicle's resulting change in position.
We trained the considered RNNs in 3000 epochs, consisting of 2000 control steps each. We applied back-propagation through time every 50 iterations and Adam as the weight adaptation mechanism (Kingma and Ba, 2014). The learning rate was annealed, such that η = 1.E − 3, η = 1.E − 4, η = 1.E − 5 during the first, second, and third 1000 sequences (first and second moment smoothing factors were set to the standard values β 1 = 0.9, β 2 = 0.999). Each vehicle was simulated for 2000 time steps (i.e. one epoch), after which the hidden state of the RNN was reset to zero and a new vehicle was initialized.

REPRISE Performance
To evaluate the robustness and abilities of REPRISE, including all relevant settings, we contrast the resulting control performance of the RNN with 36 hidden neurons with the LSTM with 16 hidden units, which have approximately an equal number of weights (1692 versus 1680, respectively). Each network was tested to reach a sequence of 50 uniformly randomly positioned targets within a centered inner area of size 1.5 × 1.5 units of the rectangular space of size 3 × 2 units. Thereby, the simulation is divided into a sequence of discrete 'events', where the agent 'becomes' one of the vehicles φ i for a continuous series of 150 timesteps. One of the agent's tasks is to infer which of these events is under way at any given time. The values in the tables below are averages over the 20 independently trained networks and 50 considered targets, whereby the target positions and vehicle succession were the same for all runs.
We applied Adam in all inference processes. Prospective inference was always P = 7 steps into the future, executing the inference cycle p = 20 times. Detailed evaluations were run contrasting different learning rates η c and η σ for the retrospective context c and system state σ inference. In our standard setting, retrospective inference covered R = 20 time steps into the past, while r = 20 inference cycles were performed. Note that during optimization the motor commands and the context inputs were clamped to their value range [0, 1], and the neural hidden states σ were clamped in accordance to the range of the respective neurons' activation function. Figure 3 shows typical flight sequences generated by an LSTM controlled by the REPRISE algorithm, in ten iteration steps. Although glider and rocket initially slightly overshoot the target, they quickly zoom in. For the stepper, the projected path is less direct, which is probably partially the case because the goal is simply not directly reachable in seven steps. It should be noted that although the images suggest that the motor effort while staying at the goal is minimized, this is not always the case, as there is currently no incentive in the system that stresses motor effort minimization.
Tables 1 and 2 show the average distance to the goal location that remained after 150 time steps, that is, control iterations. The first row of results shows the performance when the context bits are set to the correct values (no state inference), resulting in a (much simpler) active motor inference problem. The next four rows show the performance of REPRISE with different learning rate combinations, without providing the context bit values. Clearly, overly large or small values yield mediocre performance.  However, it appears that quite a large value range yields robust target reaching behavior. Consistently the best setting is with η c = .01 and η σ = .001, adapting the context bits ten times faster than the hidden states, which is most likely the case because without proper context inputs, overly fast hidden state estimations will lead to unstable adaptations. Thus, most robust performance is reached when both context and hidden state activities are adapted, yielding performance that is actually competitive -in the RNN case even superior -to the one when the context information is provided! In sum, with sufficiently small state inference learning rates η σ , the additional state inference (much harder problem, no context bit information provided) does not affect performance in a negative manner! Table 1 additionally shows the performance differences when focusing in on the three different vehicle types in the LSTM case. While the parameter dependencies are very similar, it appears that the rocket was hardest to control, most likely due to the fact that gravity needs to be continuously counteracted. Table 3 shows for the LSTM case that the average distance to the target object over the 150 steps averaged over all vehicles is smallest when the context information is provided. This was indeed the case for all three vehicles (not shown). This is expectable as context inference inevitably yields erroneous behavior during the first control steps, confirming that the switch in the vehicle identity causes initial disruptions, which are quickly stabilized.  As a final evaluation result, Figure 4 shows the inferred context input activations for the three vehicles, contrasting again LSTM with RNN performance. Clearly, the LSTM architecture is better-suited to infer the underlying control system, although the estimates are still far from optimal. It was observed that once the goal has been reached, the estimates sometimes drifted off towards more incorrect estimates -probably because the sensorimotor information was not sufficiently informative. This observation in particular suggests that both context estimation stability could be enforced, switching only when error signals suggest to do so, and active motor inference may be further optimized to maintain high context estimation certainty (Friston et al., 2015), thus generating motor commands that minimize uncertainties in the model states estimates σ.

Towards Emergent Event Encoding
The experimental setup is identical to the one detailed above, except for the fact that no contextual information is provided at any point, neither about which vehicle is currently being controlled nor about a vehicle switch or about distinct vehicle identities. As a result, context state inference needs to be applied during learning as well.

Control Performance
Again, we trained ten networks independently (with different weight initializations) on the task. During training, vehicle switches occurred every V steps (here V = 205). Context vector c was updated with a rate and depth of R c = 2 steps, repeating this retrospective adaptation r = 5 times with a learning rate of α c = .1. Weights were updated with a learning rate of α w = 10 −4 and x a x is R w = 30 steps in depth. Moreover, every 2000 steps, all neural activities in the network were reset to zero in order to avoid uncontrolled activity increase. During testing, we generated 100 successive goals (e.g. epochs), which were randomly placed in an inner reachable space of size 1.5 × 1.5. Typically, we admitted G = 150 steps to reach the goal and switched the vehicle concurrently with the goal switch (V = 150). Figure 5 shows the context guesses during a flight sequence controlled by REPRISE.
When testing active inference-based model-predictive control, the Euclidean distance to the goal locations averaged over all 100 goals and all 10 networks reached a value of e = .0027 with an adaptation rate of η c = .1. Nearly the same error value was reached when η c = .01, while the distance increased to e = .0057 and e = .1406 with η c = .001 and η c = .0001, respectively. The reached distance is clearly smaller than (i) when context inference is switched off completely, which yields e = .0998, (ii) when the learning rate during model learning is lowered to α c = .01 yielding e = .0200, or (iii) when the vehicle switch occurs more frequently, e.g. every five steps, than the context adaptation, e.g. R c = 20, yielding e = .0429. On the other hand, when the vehicle switch V during learning occurs randomly between every 20th and 30th time step and R c = 2, a comparable error of e = .0032 is achieved.
Interestingly, as shown above, we achieved a best minimal error of e = .006 when the context values were set to one-hot vectors during training, which is larger than the value e = .0027 in these results. This implies that the network developed contextual state indicators during learning that are more suitable for the task than the one-hot vectors. Moreover, the results imply that a reasonable wide parameter range yields comparable performance. However, context state adaptations need to occur more frequently than vehicle switches (i.e. event changes) and need to be adapted with a sufficiently large adaptation rate, e.g., α c = .1.

Current Controlled Vehicle Inference
Seeing the improved control performance when context inference is switched on (no one-hot vectors), we analyzed the development of the context state estimates during testing further. Here we focus on reporting the results for one of the ten networks, which serves well for illustrating the main points. Qualitatively, the results look similar for the other networks. Figure 6 shows the adapted context values c before a goal switch, that is, at the end of each of the 100 epochs, colored according to the vehicle that was just controlled. The results show three observable clusters, indicating that the network has learned to separate the sensorimotor dynamics of the three vehicles into distinct but somewhat overlapping clusters.
Further analyses revealed that the context state estimates can drift severely once the target is reached. This is particularly the case because in the setup Stepper and Glider essentially just need to remain still at the goal state, which they can achieve by sending out zero motor commands but also by sending out equally strong ones to all four motors. Similarly, the rocket can mimic gravity by its ineffective downwards thrust motors. Thus, particularly at the goal, the sensorimotor signal about which vehicle is currently controlled can become ambiguous. Figure 5 shows such an exemplary case, while the stepper is controlled. During the initial steps, the signal from the previous vehicle still influences the gradient. Then, the signal tends approximately towards the preferred center of the stepper's context state vector. After reaching the goal, however, a drift of the vector can be noticed.
Looking at the previous figures, one can notice that the network is easily identifying the Rocket (in red), but sometimes gets confused between the Stepper (in green) and the Glider (in blue). To get a closer look at this effect, we have calculated the Euclidean distances between the centers of the clusters for the ten independently trained networks. The relative distance between the center of the Rocket cluster and the center of the Glider cluster was the largest on average (d=.415), followed by the distance from the Rocket to the Stepper (d=.348), while the distance between Glider and Stepper was the shortest (d=.237). The gravity effect on the Rocket and the lack of two of the motors seems to require the most distinct encoding. On the other hand, the distance between Glider and Stepper is the smallest on average, probably due to the fact that both have 4 motors and do not experience gravity.

System Dynamics
Running the network in a "context free" manner did not only raise the question of how the context estimates change over time and where to they converge, but also how these dynamics correlate with goal reaching behavior and the unfolding sensorimotor prediction error dynamics. In particular, if further conceptual abstractions of the sensorimotor dynamics are to be fostered, an event boundary signal in the form of a measurable, significant increase in prediction error (akin to Gumbsch et al., 2017a) upon vehicle change would be very useful. Figure 7 shows exemplary results for ≈ 750 time steps, plotting the Euclidean distance between the vehicle and the target, the current prediction error, as well as the corresponding trajectories of the vehicle estimates, colored and marked by the currently controlled vehicle (unknown to the system). It is easy to see how the prediction error strongly increases after each vehicle and goal switch (V = G = 150). Moreover, the error decreases when the goal is reached and then stays rather constant at a low but not necessarily 0 level. The context state estimates furthermore confirm the reported drift behavior. Moreover, particularly the second time the rocket is controlled, the context state estimates reach extreme values. Gradients that go beyond their boundaries of [0, 1] are currently ignored. Future versions should consider continuously differentiable context state activation functions, rather than inducing hard boundaries.   Figure 8 shows typical prediction error dynamics when vehicle and goal switches (V = 100, G = 150) occur asynchronously. Thus, error dynamics may be used to detect surprise signals, which can suitably indicate event boundaries (here vehicle switches) (Butz, 2016;Gumbsch et al., 2017a).

Summary, Conclusions & Future Perspectives
We have shown that REPRISE maintains ANN activity that reflects on the past and projects its own state into the future, thus continuously optimizing its internal (generative) state estimates about its own body and the environment as well as imagined upcoming environmental interactions. We have developed this system as a first step towards sensorimotor-grounded, event-oriented abstractions. Essentially, the context vector c signals the contextual "event" the system is currently in. We have shown that the event was inferable after model learning but that suitable event encodings can also emerge during learning when inference is applied to both, contextual estimates and model weights.
In particular, the results confirm that REPRISE is able to identify the currently controlled vehicle without ever being informed about a vehicle switch, the vehicle identities, or that there are three types of vehicles. As long as the vehicle type is switched less frequently than the context state estimates are adapted and as long as the learning rate is sufficiently large, very good control performance was achieved. Moreover, distinct contextual encodings developed for each vehicle in a self-organized manner. The resulting network structure could be used to minimize prediction error and distance to target, concurrently with deducing the currently controlled vehicle, by performing prospective, goal-directed behavior while retrospectively inferring appropriate contextual states.
Elsewhere, Butz (2016Butz ( , 2017 has proposed that predictive encodings may be suitably compressed into stable event and event boundary codes, yielding conceptual abstractions. REPRISE achieves this for the first time in an RNN-based control architecture. However, here we have focused on analyzing sensorimotor dynamics, and thus distinct events characterized by their dynamic regimes. We believe that similar event encodings may be developed in other scenarios, such as when manipulating objects, using tools, or even forming place fields for navigating through an environment. In all cases, it appears that particular sensorimotor regimes apply, especially when encoding objects in manipulator-relative frames of reference (Calinon et al., 2010;Gumbsch et al., 2017b). To foster the development of such encodings further, we believe that unexpected changes in prediction errors should be used as an indicator signal for an event change (Butz, 2016;Gumbsch et al., 2017b). The gathered results in this respect suggest that these signals are indeed available in the architecture.
Once the automatic learning of event encodings is achieved, event-predictive cognition on the compact event-encoding level will become possible, potentially offering a step towards conceptual and compositionally re-combinable event schema abstractions. As an additional challenge, it should be kept in mind that the implemented processes currently fully rely on error backpropagation through time. Implementations of probabilistic inference processes along similar lines are well-imaginable.
Moreover, events may be compressed further when particular trajectories and dynamics become important for achieving particular goals, such as when opening a door or steering a car. The addition of event-specific control routine optimization techniques seem to be within the grasp of REPRISE. The techniques may focus on achieving particular event transitions as goals and may be implemented by means of policy gradient techniques (Stulp and Sigaud, 2013), In this case, but also for improving system scalability and focusing learning in general, the identified event boundary signals may be particularly useful.
Finally, we believe that the exploration of deeper hierarchies, akin to the networks discussed in Tani (2017), but with slower, event-adaptive dynamics in deeper levels of the hierarchy, constitutes a highly important next research step. By focusing RNN learning further on predicting the occurrence of event transitions, it may be possible to develop conceptual abstractions that can be suitably linked with linguistic structures that verbalize executable environmental interactions (Schrodt et al., 2017).